Chunking Strategies
Fixed, recursive, semantic and section-aware chunking. The #1 lever for RAG quality.
What is chunking?
Chunking splits a long document into smaller units (chunks) that each fit in the embedding model and the LLM context. The chunk is the unit of retrieval — what you embed, what you search, what you return as context.
Chunk size and strategy directly determine retrieval quality. Too small → loses context. Too large → dilutes relevance and wastes context budget. Wrong boundaries → splits a concept across chunks.
Why chunking is the #1 lever
Most RAG quality problems are chunking problems, not embedding or model problems. A great embedding model on bad chunks performs worse than a mediocre model on well-chunked text. Chunking is cheap to change, easy to A/B, and high-impact. Always start here when retrieval is poor.
Chunking in the pipeline
Ingestion → Extract text → CHUNK → Embed → Store. Chunking sits between extraction and embedding. The chunk is what gets embedded, so the chunk boundary defines what 'similar' means. Section-aware chunking keeps a heading with its content; semantic chunking groups by meaning; fixed-size just splits by character count.
Three chunking strategies
def fixed_size_chunk(text: str, size: int = 512, overlap: int = 64) -> list[str]:
"""Naive: split by character count with overlap."""
chunks = []
i = 0
while i < len(text):
chunks.append(text[i:i+size])
i += size - overlap
return chunks
# Problem: splits mid-sentence, loses structure, no section awarenessExperiment: chunk size vs retrieval
Vary chunk size and overlap. Observe the trade-off between context coherence and retrieval precision.
What to observe
Smaller chunks → higher precision but lower recall and more embeddings cost. Larger chunks → higher recall but diluted relevance and wasted context. Overlap bridges concepts split across boundaries. 512/64 is the production default — measure on YOUR data before deviating.
Production chunking
Production chunking: section-aware (use document structure), overlap to bridge boundaries, metadata (section, position, source) on every chunk, and a chunk-eval harness that measures recall@k and context_relevance per strategy. Version your chunking — it's part of your retrieval system, not a one-time choice.
Challenge
A policy document has a numbered procedure split across chunks — step 3 ends chunk A, step 4 starts chunk B. Retrieval returns chunk A but the answer needs steps 3-5. How would you redesign chunking to keep procedures intact?
Production checklist
Production checklist
0 of 6 checked
Knowledge check
Your RAG recall is good (0.88) but precision is poor (0.61). Top-k returns relevant-but-fragmentary chunks. What is the most likely fix?
Complete
You can now choose and tune chunking strategies. Next: embeddings and vector search — how chunks become searchable, and how to measure it.
Mark this chapter as complete
Track your progress and unlock the next chapter.