Module 12: Private document search (RAG), fully offline · Lesson 12.3 · about 25 min
Cutting documents into chunks
Cut long documents into pieces small enough to search and big enough to make sense. Choose chunk size and overlap with a calculator, and write a 10-line chunker in Python.
Module 12 · about 25 minutes · read first: 12.2. You’ll be able to choose a chunk size and overlap, and cut any text into chunks with Python.
1. Why this matters
Chunking is the most underrated setting in RAG. The same model and the same files can give good or poor answers just from how the documents were cut. It is also the setting you can tune for free.
2. Too small, too big, just right
- Too small (one sentence): each chunk is precise but loses its context. “It must be signed within 30 days” no longer says what it is.
- Too big (whole chapters): each vector blurs many topics together, search gets vague, and five of them may not fit in your context window.
- Just right: usually 200 to 500 words (about 250 to 650 tokens), one idea or a few paragraphs, with 10 to 20% overlap.
Think of cutting a long photo strip into frames for an album index. Tiny frames show sharp details but no scene. Huge frames show the whole party, but the index card can only say “a party”.
Better still, cut along the document’s own seams: headings, then paragraphs, and only then by length. Keep each chunk’s file name and page number beside it: you need them for citations.
3. A chunker in ten lines
Save this as chunker.py in a new folder called askdocs. The next lessons build the rest of the project beside it.
def chunk_text(text, size=300, overlap=50):
"""Cut text into pieces of about `size` words; each piece repeats the last `overlap` words of the one before."""
words = text.split()
step = size - overlap
pieces = []
for start in range(0, max(len(words) - overlap, 1), step):
piece = " ".join(words[start:start + size])
if piece:
pieces.append(piece)
return pieces
It counts words, not tokens, because that needs no tokenizer; 300 English words is roughly 400 tokens. Test it: print([len(p.split()) for p in chunk_text(" ".join(["w"] * 700))]) gives [300, 300, 200].
The document. 700 words. Settings: size=300, overlap=50, so each new chunk starts 300 − 50 = 250 words after the last.
Chunk 1. Words 1 to 300.
Chunk 2. Starts at word 251 and runs to 550. Words 251 to 300 are in both chunks: that is the overlap. A sentence cut at word 300 appears whole in chunk 2.
Chunk 3. Starts at word 501 and runs to the end, word 700, so it holds 200 words. The loop stops: the next start (751) is past the end.
The count. (700 − 50) ÷ 250 = 2.6, rounded up to 3 chunks, matching [300, 300, 200].
Maths and principle
Principle: chunk size trades precision of each match against context carried by each match; overlap guards the seams.
Equation: for a document of words, chunk size and overlap ,
(⌈ ⌉ means round up). Prompt sources ≈ k × s.
Analogy: cutting a long photo strip into frames for an album index. Tiny frames show sharp details but no scene; huge frames show the whole scene but the index card for each can only say “a party”. Overlap is printing the edge of each frame on the next, so nobody’s face is cut in half. Where it breaks: tables and code do not have word-level meaning; cut them by rows or functions instead.
Try it: a 10,000-word report, s = 300, o = 50. About how many chunks? ((10,000 − 50) ÷ 250 ≈ 39.8, so 40.)
- W
- the number of words in the document
- s (chunk size)
- the most words in one chunk
- o (overlap)
- how many words each chunk repeats from the end of the one before
- s − o (step)
- how far along the text each new chunk starts
- Physical meaning
- Chunking is cutting a long scroll into index cards. Each card should hold one idea so its label is sharp, and the edges overlap a little so no sentence is torn in half. More overlap means more cards and a bigger index, but fewer facts lost at the seams.
“Bigger chunks give the model more context, so they are always better.” One vector has to stand for the whole chunk. Mix ten topics into it and it matches none of them well, and five big chunks may not fit in your context window. Test sizes with your own questions (Lesson 12.7); 200 to 500 words is a good start.
4. Worked example
Made-up numbers. An HR team tests three settings on 50 known questions (the method is in Lesson 12.7). 100-word chunks: right passage in the top 5 for 33 of 50 questions. 300 words with 50 overlap: 44 of 50. 1,000 words: 36 of 50, and prompts three times longer. They keep 300/50.
5. Practice
- Chunk a document of 1,000 words with size 300 and overlap 0, then with overlap 100. How many chunks each time?
- Why might legal contracts like a slightly larger chunk size than chat logs?
- Change
chunk_textso it first splits on blank lines (paragraphs) and only cuts paragraphs longer thansize.
Answers
- Overlap 0: 4 chunks (300, 300, 300, 100). Overlap 100: 5 chunks (step 200).
- Clauses are long and refer back within the same section; chat messages are short and self-contained.
for para in text.split("\n\n"):thenpieces += chunk_text(para, size, overlap) if len(para.split()) > size else [para.strip()], skipping empty paragraphs. Very short paragraphs can be joined to the next one.
New words: chunk size (how long each piece is), overlap (words shared between neighbouring chunks), top-k (how many chunks the search returns).
- Chunk size trades a sharp match (small) against enough context to make sense (big); 200 to 500 words is a good start.
- Overlap of 10 to 20% keeps facts that straddle a cut whole in at least one chunk.
- Cut along headings and paragraphs first, and only then by length; tables and code by rows or functions.
- Number of chunks = ⌈(W − o) ÷ (s − o)⌉; prompt sources ≈ k × s.
- Keep each chunk’s file name and page beside it for citations.
Check yourself
Sources
- LangChain text splitters, a common reference implementation, https://python.langchain.com/docs/concepts/text_splitters/
- Pinecone, Chunking Strategies for LLM Applications, https://www.pinecone.io/learn/chunking-strategies/
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.