Chunking

The decision everything downstream inherits

Retrieval can only ever return a chunk you created. If the sentence that answers the question was split across two chunks, no embedding model, no re-ranker and no amount of prompt work will put it back together. This is why chunking deserves more attention than it usually gets: it is the one stage whose mistakes are unrecoverable later. It is also the stage most often left at a library default — a thousand-character fixed window with two hundred characters of overlap is a reasonable starting point and a poor finishing point for most real corpora.

The three strategies, worked through on one paragraph

Take a paragraph explaining a refund policy: a sentence stating the policy, a sentence giving the time limit, and a sentence listing exceptions. Fixed-size splitting cuts at a character or token count regardless of meaning — if the cut falls mid-sentence, between the time limit and the exceptions, a query asking about exceptions retrieves a chunk that states the limit but not the exceptions, or vice versa, and the answer is half-true. Recursive splitting respects the document’s own structure, preferring to cut at paragraph breaks, then sentence breaks, then only falling back to a raw character count if nothing structural is available nearby — so the refund paragraph, being one coherent block, is far more likely to survive intact or to split along a sentence boundary that still leaves each piece meaningful on its own. Semantic splitting goes further still, embedding individual sentences and cutting wherever consecutive sentence similarity drops sharply, directly detecting a topic change rather than inferring it from formatting — valuable for unstructured transcripts or scanned prose that has no headings or paragraph marks to exploit, at the cost of an embedding pass over the entire corpus just to decide where to cut.

Size and overlap, with concrete numbers

A chunk in the range of roughly 256 to 512 tokens is a common, reasonable middle ground: small enough that an embedding represents one coherent idea rather than several blended together, large enough to retain the surrounding sentence or two of context a single fact usually needs to remain interpretable on its own. Overlap of ten to twenty percent of the chunk size — fifty to a hundred tokens on a five-hundred-token chunk — exists so that a sentence sitting near a boundary appears intact in at least one chunk even if it is split in its neighbour, at the direct cost of storing some duplicated text and later having to deduplicate near-identical results that both legitimately matched.

Chunking different content types

Plain prose chunks reasonably well by sentence and paragraph structure alone. Code should be chunked along function or class boundaries rather than by character count, since splitting a function body mid-statement produces a chunk that is syntactically meaningless on its own. Tables are often best kept whole or chunked row-group by row-group with the header repeated in every chunk, since a table row without its column headers loses its meaning entirely. PDFs carry their own hazard: text extraction frequently loses the original layout, so a two-column page can interleave unrelated sentences from each column into one garbled stream before chunking ever begins — a multi-column PDF is worth checking by eye at least once before trusting its extracted text.

An advanced pattern: small chunks, larger parents

Parent document retrieval splits a document into small chunks purely for the purpose of matching — small chunks embed more precisely, because they represent one narrow idea — but when a small chunk is retrieved, the system hands the model the larger surrounding section it came from, rather than the tiny matched piece alone. This captures the best of both sizes: precise matching from small chunks, and full surrounding context in what actually reaches the model.

Measuring whether your chunking is any good

The only honest test is retrieval recall: take real questions, mark which chunk should answer each one, and measure how often that chunk actually appears in the retrieved set. A chunking strategy that looks reasonable on inspection but scores poorly on this measurement is the one to revisit before touching anything downstream in the RAG pipeline.