Semantic chunking
A document-splitting technique that uses embeddings to detect semantic boundaries — produces more coherent chunks for RAG than fixed-size chunking, improving retrieval quality.
Chunking is the unglamorous foundation of RAG quality. Fixed-size chunking (split every N tokens) splits sentences mid-thought; semantic chunking detects natural boundaries by measuring embedding similarity between adjacent sentences. Where similarity drops below a threshold, a new chunk starts.
In 2026 semantic chunking is increasingly the default for RAG production stacks. LlamaIndex, LangChain, and several vector databases ship with semantic chunkers. The improvement: 10–20% lift in retrieval quality vs naive fixed-size chunking, with minimal additional cost at index time.
For agent builders, semantic chunking matters most when your source documents have varied structure (mixed paragraphs, lists, headings) and your queries are conceptual. For uniform documents with simple queries, fixed-size chunking is often fine.
Frequently asked
When does semantic chunking matter most?+
When source documents have varied internal structure (mixed prose, tables, lists). When queries are conceptual rather than keyword-matching. For both, semantic chunking lifts RAG quality meaningfully.
How expensive is semantic chunking?+
A few cents per 100K tokens of source documents at index time. Free at query time (just retrieves from pre-chunked index). The cost is trivial relative to the quality lift.