Part 4 of 6 · 3 min read
Size, overlap and metadata
Choose by measuring, and keep the heading path with each chunk.
Choosing size and overlap
| Question type | Lean toward | Why |
|---|---|---|
| "What is the default port?" (one fact) | smaller chunks | a precise query matches a precise piece |
| "Explain how the guest network works" (a topic) | larger chunks or whole sections | the answer is spread over several sentences |
| Code, tables, Q&A lists | chunk by the natural unit (function, row, pair) | cutting these destroys meaning |
Practical defaults, to be tested rather than trusted: 200 to 800 tokens per chunk, 10 to 20 percent overlap, and a heading prefix when you have headings. Smaller chunks also mean more vectors to store and search, so cost rises as size falls.
A pattern that gets both precision and context is small-to-big: index small chunks, but hand the model the larger parent section. Hard exercise 3 builds it.
Metadata: the part that makes answers checkable
Every chunk should carry source, path (headings), and where available page and url. You need them for three things later in this level: showing citations (module 5), filtering ("only the 2026 manual"), and refreshing an index when a document changes (module 9). Adding them at chunking time is easy; reconstructing them afterwards is painful.
Common mistakes
- Tuning chunk size without questions to measure on.
- Indexing parsed text you never read.
- Splitting tables and lists as if they were prose.
- Using the same settings for code, manuals and chat logs.
- Changing the tokenizer or chunk size and forgetting to re-index; old and new chunks then mix.
Interview angle
"How do you chunk documents for RAG?" A weak answer is "split into 500 tokens". A strong one starts with the questions: what do users ask, and what unit holds an answer? Then: parse first and inspect the output, prefer structure-aware splitting with a heading prefix, add modest overlap for prose, keep metadata, build a small labelled question set, and choose size by measuring answer-intact and top-k rates. Mention small-to-big retrieval and that changing chunking means re-indexing.
Check yourself
A chunker cuts "Set the timeout to 90" and "seconds in the admin page" into two chunks. Which of the numbers in section 4 would notice, and which would not? (Intact notices, because the whole answer is in no chunk. The ceiling for top-1 and top-3 then drops too.)
For the curious Two bugs found in the example code
The PDF cleaner crashed on a blank page, because it looked at the first and last line of a page that had none. It also counted a one-line page twice, so a lone line was dropped as a "repeated header". Both are fixed with tests. The report also divided by zero when given no questions. The cleaner still joins a real hyphenated word that happens to break at a line end ("dual-" and "band" become "dualband"), which is why you read the output.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Read parsed output | Open ten parsed documents and look for lost tables and junk before indexing anything. |
| A small labelled question set | Ten to fifty questions with exact answers, kept next to the code. |
| Document loaders and splitters in a library | Frameworks ship recursive and Markdown-aware splitters; you configure size and overlap. |
| Re-index on change | Changing chunk size or parser means rebuilding the index. |
Draft: confirm which loader and splitter libraries your team uses before relying on this.