Part 5 of 6 · 4 min read
Exercises
Easy
Easy 1: Run and read (20 minutes)
In code/mini-fresh, run pytest and python demo.py. From section 3 of the demo, find the strategy with the lowest mean accuracy and the one with the highest embed count. For the lowest one, say which of stale, ghost, missing and conflict it suffers from, with the percentages.
Done when: you name two strategies from the table and give two or more percentages for the first.
Hint
the lowest mean is the frozen index; the highest embed count is the daily rebuild.
Easy 2: Name the symptom (15 minutes)
For each report, name the type (stale, ghost, missing or conflict) and a likely cause: (a) "the bot still lists the old refund window"; (b) "it quotes a policy we withdrew in March"; (c) "it can't find the new onboarding guide"; (d) "sometimes it says 30 days and sometimes 60".
Done when: you have four labels and four causes, and you say which two share a root cause.
Hint
(a) and (d) can share a cause; look at what is kept rather than what is added.
Easy 3: Pick the detector (15 minutes)
For each case pick modified time, document hash or chunk hash, with a reason: (a) 5 million files on a network drive where reading all of them nightly is too slow; (b) 300 long contracts where each edit changes one clause; (c) a wiki where a bot rewrites every page's footer date nightly; (d) files restored from backups with their original timestamps.
Done when: you have four picks, each with a stated weakness of your choice.
Hint
think about both cost (reads) and what the method can miss.
Medium
Medium 1: Normalise before hashing (45 minutes)
In the example, add a line of trailing whitespace or a changed "last viewed" footer to one paragraph and show that it triggers a pointless re-embed. Add a normaliser (collapse whitespace, strip the footer) used by split before hashing, and prove with a test that cosmetic edits cause 0 embeds while real edits still cause 1.
Done when: both tests pass and you list two normalisations that would be wrong (for example, lowercasing a case-sensitive code).
Hint
normalise before computing the chunk id, and test that two texts differing only in whitespace share an id.
Medium 2: Add expiry (60 minutes)
Give some chunks an expires_at day and make search ignore expired chunks and sync purge them. Add a "promotion" fact that expires on day 14 and show that a frozen index answers with it on day 20 while yours does not.
Done when: tests cover before and after expiry, and you explain what happens to a question whose only answer expired.
Hint
an expired chunk should leave the index only at the next sync; the search filter covers the gap.
Medium 3: Shrink the window (60 minutes)
Change simulate so that a sync can also run right after each day's changes (before questions), then compare lag-0, daily and weekly. Report mean accuracy and the embed and read counts.
Done when: you have a three-row table, and you state what the extra frequency cost in reads and in embeds.
Hint
lag 0 needs the sync to see the day's edits, so apply the changes, sync, then grade.
Hard
Hard 1: A real embedding model and vector store (150 minutes)
Replace the BM25 index with a real embedding model and a vector database of your choice. Keep content-addressed ids. Run an edit, a delete and an add and confirm that only new chunks are embedded. Then switch to a second embedding model both ways: mixed (half the vectors) and blue-green, measuring answer accuracy on 30 questions. (I did not run this, so your numbers are the ground truth.)
Done when: you have a table of accuracy for mixed and blue-green, the count of embed calls for each sync step, and your configuration-id check in code.
Hint
record the embedding model name with each vector and compare it with the query side before searching.
Hard 2: Versioned index with 'as of' questions (120 minutes)
Keep superseded chunks with valid_from and valid_to instead of removing them, and add ask(question, as_of=day). Test it on the 30-day simulation: "what was Orion's seat limit on day 6?" must return the value that was true then.
Done when: you answer 20 as-of questions from the simulation's history correctly, and describe the storage cost versus the nightly-incremental index.
Hint
set valid_to instead of removing; a query "as of" a day filters on valid_from <= day < valid_to.
Hard 3: A sync that survives bad inputs (120 minutes)
Make the source unreliable: sometimes a listing is empty, sometimes a read fails, sometimes a document is half-written. Extend sync_chunk_hash so that none of these damages the index (retries, partial-failure handling, per-document transactions, the guard). Then build a monitor that reports lag, blocked runs and documents skipped.
Done when: a test with 20% random read failures never leaves the index worse than before the run, and the monitor output shows the failures.
Hint
write the new index state to the side and swap it in only when the document finished cleanly.