A chunk is the unit your retriever can find and your model can read. Too small and it loses context; too big and it drowns the answer. Start structural, add overlap, measure — and only then look at fancier methods.
Every RAG tutorial spends three paragraphs on chunking and thirty on vector databases. In our client projects it is almost always the other way around: the database is a commodity, the cutting strategy is where the quality is won or lost. This article is the ruler-and-scissors version. No embeddings yet; those come in part 2.
The problem in one picture
You have a 40-page document and a question. The model can read maybe 3 pages’ worth of text comfortably. Something has to decide which 3 pages — and what counts as a “page”. That something is chunking. Drag the slider and watch how the same text gets cut differently.
Fixed-size chunks
The baseline: cut every N tokens, no questions asked. It is fast, predictable and it is what most frameworks default to. Its failure mode is visible in the figure above at small sizes — sentences are sliced mid-thought, and the chunk containing “the contract terminates” may not contain “unless notice is given 90 days before”.
def fixed(tokens, n):
return [tokens[i:i+n] for i in range(0, len(tokens), n)]Overlap
The cheapest fix: let consecutive chunks share a tail. An overlap of 10–20 % means a sentence cut in half at the end of chunk k appears whole at the start of chunk k+1. You pay for it in index size (more chunks) and in duplicated retrieval results, which you will need to de-duplicate later.
Overlap is not a strategy; it is a bandage on a strategy. Use it, but know what it is covering.
Cutting along structure
Documents already have cuts in them: headings, paragraphs, list items, table rows. Cutting there keeps units that a human would consider “one idea”. Tick the checkbox in Fig. 1 to see the difference. In practice this means parsing Markdown/HTML/DOCX structure first and only falling back to fixed-size inside very long sections.
- Headings → prepend the heading path to each chunk (“Contract › Termination › Notice”). Retrieval quality jumps noticeably; we measured +6 points recall@5 on a legal corpus.
- Tables → never split a row from its header. Serialize rows as “header: value” pairs.
- Code → cut on function/class boundaries, not lines.
Semantic chunking
The fashionable option: embed sentences, cut where the embedding shifts. It is elegant and sometimes better — but it is slow, it depends on the embedder you have not chosen yet, and in our tests it rarely beats good structural chunking on well-formatted documents. Where it shines: transcripts, emails, anything without structure to lean on.
How to pick
Structural first. Overlap 10 %. Chunk target 200–400 tokens for Q&A, 600–1,000 for summarisation. Then measure with 30 hand-written questions before touching anything else. Chunking changes that don’t move that number are noise.
Try the strategies on your own PDF in the playground below — it runs on our infrastructure, no sign-up needed for this one.
Next in the series
- Chunking, explained with a ruler and a pair of scissors
- Embeddings: what close means and why your model disagrees
- Hybrid search — BM25 is not dead
- Reranking, the step that actually moves accuracy
- Evaluating RAG with 30 questions and a spreadsheet
- Updating the index without breaking everything