RAG Chunking Strategies: How to Split Documents for Better Retrieval

When a retrieval-augmented (RAG) assistant gives a vague or wrong answer, people usually blame the model. More often the real culprit is upstream and invisible: how the documents were split. Chunking decides what pieces of your content are eligible to be retrieved, and therefore what the model actually gets to read. Get it wrong and the best model in the world will confidently answer from the wrong paragraph — or half of one.

Why chunking decides RAG quality

Retrieval works by matching a user's question against small pieces of your documents, then handing the top matches to the model as context. The chunk is the unit of retrieval — the model never sees your document, only whichever chunks were pulled. So chunk boundaries quietly set the ceiling on answer quality:

• If the sentence that answers the question is split across two chunks, neither matches well and the right answer may never surface.
• If a chunk mixes three unrelated topics, its embedding is muddy and it competes poorly against cleaner chunks.
• If retrieved chunks are bloated, they crowd out the good context and eat into the context window.

Bad chunks produce bad answers even with a perfect model and prompt — which is why this is worth tuning before anything else.

Chunk size: the core trade-off

Chunk size is the first dial, and it cuts both ways:

• Too small — a chunk holds a sentence or two with no surrounding context. Retrieval gets precise on keywords but the model loses the thread: the pronoun whose antecedent lived in the previous sentence, the caveat in the next one.
• Too large — a chunk spans several ideas, so its embedding averages them and matches many queries weakly and none strongly. You also pay in tokens and dilute the relevant sentence inside a wall of text.

There's no universal number, but a useful rule of thumb is a few hundred tokens per chunk — large enough to hold a complete thought, small enough to stay on one topic. Dense reference material and short FAQs lean smaller; narrative or explanatory prose can go larger. Treat any starting size as a hypothesis to test on your own data, not a setting to lock in.

Overlap: don't split an idea in half

Overlap means each chunk repeats a little of its neighbor's text — say the last sentence or two of the previous chunk. It's cheap insurance against the worst failure mode: an idea cut exactly at a boundary so that no single chunk contains it whole. A modest overlap (often around 10–20% of chunk size) keeps continuity across boundaries. The cost is some redundancy — the same text stored and retrieved twice — so it's a small tax you pay to avoid orphaned context, not a dial to crank to the max.

Strategies, and when to use each

• Fixed-size — split every N tokens or characters, no regard for content. Simple, fast, and a fine baseline, but it happily cuts mid-sentence. Best when documents have little structure and you just need a starting point.
• Recursive / character — split on a priority list of separators (paragraphs, then sentences, then words), falling back only when a piece is still too big. This respects natural boundaries while keeping sizes in check — a strong, general-purpose default for mixed prose.
• Sentence or paragraph — split on real linguistic units. Good when content is cleanly written and each paragraph holds one idea, though sizes vary and very long paragraphs may need a secondary split.
• Structure-aware — use the document's own scaffolding: Markdown headings, sections, list items, table rows. Ideal for docs, wikis, and technical content where a heading marks where one topic ends and the next begins, and keeping a heading with its section gives each chunk context.
• Semantic chunking — group sentences by meaning, starting a new chunk where the topic actually shifts (via embedding similarity) rather than at a fixed length. This produces the most coherent chunks and shines on long, meandering documents, but costs more to compute — reach for it when retrieval quality matters more than indexing speed.

How to choose

The right strategy is the one that matches your content and your model:

• Match size to the content. Reference tables and FAQs want tight, self-contained chunks; long-form explanation wants roomier ones.
• Respect the document's structure. If your source has headings, sections, or rows, split on them before falling back to arbitrary lengths — structure a human uses to navigate is structure retrieval can use too.
• Match size to the embedding model. Every embedding model has an input limit and a range where it represents text best; chunks that overflow it get silently truncated, so keep them comfortably inside that window.
• Retrieve few, good chunks. The goal isn't to stuff the prompt but to surface a handful of clean, on-topic chunks. Well-formed chunks let you retrieve fewer and still cover the answer.

Common mistakes

• Splitting blindly by character count, slicing sentences, tables, and code in half.
• Zero overlap, so boundary-straddling ideas vanish from every chunk.
• One chunk size for everything — a legal contract and a support FAQ don't want the same treatment.
• Stripping structure (headings, lists, tables) during ingestion, discarding the very signals that make chunks coherent.
• Never inspecting the output — shipping a config without looking at what the chunks actually contain.

Iterate before you scale

Chunking is empirical. The fastest way to improve a RAG system is usually to look at the chunks, not the prompt — see where boundaries fall, spot the split-in-half ideas, and adjust size and overlap before you embed the whole corpus. A quick way to build that intuition is to paste in a document and watch how chunks change as you tune the settings.

In LLMGraph, chunking, embeddings, and retrieval are handled for you as part of the workflow — point an assistant at your documents, tune how they're split, and see the effect on real answers without building a pipeline yourself. Start with a sensible default, test on your own content, and refine. The model gets the credit, but the chunks do the work.