Four ways to chunk

Four ways to chunk

Let size be the maximum characters per chunk. (Real systems often count tokens so a chunk is guaranteed to fit a model limit; the logic is the same. Level 1, module 1 explained tokens.)

3.1 Fixed size, with optional overlap

def fixed(text, size=300, overlap=0): ...

Cut every size characters. With overlap=20, each chunk starts 20 characters before the previous one ended, so a sentence cut at a boundary appears whole in at least one chunk.

• Good: trivial, predictable, fast, works on any text.
• Breaks: it cuts in the middle of words and sentences, and ignores structure completely.

3.2 Sentence packing

Split into sentences and pack whole sentences into a chunk until adding the next would exceed size. A sentence longer than size becomes its own chunk, so the limit is a target, not a guarantee.

• Good: never cuts a sentence.
• Breaks: a sentence rarely says everything. "It expires after 24 hours" is useless without the sentence before it.

3.3 Recursive splitting

Try the biggest separator first (blank line), then line break, then sentence end, then space. Recurse only into pieces that are still too long, then merge small neighbours back up to size.

• Good: respects paragraphs when it can and falls back gracefully. A common default in libraries.
• Breaks: it still does not know what a section is; a paragraph from "Guest network" and one from "Safety" can land in the same chunk.

3.4 By structure (headings)

Walk the parsed blocks, keep the heading path, and make one chunk per section, splitting a section further only if it is too long. Each chunk keeps the path and, optionally, starts with it as a prefix: Brightline R7 router > Troubleshooting > Factory reset.

• Good: chunks follow the author's own topics, and the prefix puts the topic words into the text that the retriever scores.
• Breaks: it needs reliable headings. Sizes vary a lot (a section of three words and one of three pages). Documents without headings fall back to another method.

Measuring a chunking choice

To compare strategies you need questions with known answers. The example has 12, each paired with the exact answer text that must appear whole inside one chunk:

("How long until the status light is steady green?", "which takes about 90 seconds")

and three numbers per strategy:

What the run shows (toy corpus, keyword retriever):

• At about 120 characters, fixed chunks lose answers: only 75 percent of answers are intact and top-1 is 67 percent. Overlap of 20 percent rescues some answers (83 percent intact) but not the ranking.
• At about 250 characters, every method does well, and fixed with overlap reaches 100 percent top-1. The toy manual's sections are short, so "big enough to hold a section" is easy.
• At about 500 characters, the size-based methods drop to between 83 and 92 percent top-1: a bigger chunk holds several topics, so the keyword scorer sometimes ranks a chunk about the wrong topic first. By structure stays at 100 percent because it never mixes sections.

Do not memorise those figures. The lessons that carry over are these: (1) the intact ceiling comes first, (2) bigger is not automatically better, (3) the best size depends on your questions and documents, and (4) you can only know by measuring.