Part 1 of 6 · 1 min read
Overview
In this module you will
- Name the ways an index drifts away from its source: stale, ghost, missing and conflicting results.
- Detect change three ways (modified time, document hash, chunk hash) and say what each costs and what each misses.
- Update an index incrementally: embed only new chunks, remove vanished ones, and hide deleted documents with tombstones.
- Protect an index from a failed crawl that looks like "everything was deleted".
- Plan a model change: why half-re-embedded indexes fail, and how to switch safely.
Everything in this level assumes the index describes the world as it is. It does not stay that way: policies change, pages are removed, new documents arrive, and you will one day upgrade the embedding model. An agent that confidently quotes a withdrawn policy is worse than one that says "I don't know", and the user cannot tell the difference. Freshness is also a cost problem: the obvious fix (rebuild everything every night) re-pays the embedding bill for text that did not change. This module is about getting both right.
About the examples
The "company" is invented: a help-center wiki of 12 plan documents (rising to 13 live by day 30), four one-sentence paragraphs each (seat limit, retention period, support tier, export format). Over 30 simulated days the source gets 2 edits and 2 "touches" per day, 4 new documents and 3 deletions, plus one restored backup. The embedding model is a stand-in that counts calls, and search is a small BM25. Questions are asked before each night's sync, so even the best method has a one-day window. Real sources are messier (formatting changes, renames, permissions, huge files) and real vector databases have their own update semantics. The numbers show direction, not benchmarks. No real service was called.