Presence: a plain check on text: is the correct answer string in the context the model would be shown (and no out-of-date value beside it)? It does not say whether a model would then use it.
Modelled accuracy: a made-up probability of the model answering correctly, from an assumed curve. A direction, not a measurement.
Lexical selection: choosing items by the words they share with the question, as opposed to by meaning.
For each policy and each budget from 500 to 8,000 tokens, I built the context for every one of the 100 questions and asked two things. Present: is the correct answer string in the text, with no stale value beside it? Modelled: what would my invented reader score? Here the window limit is lifted, so that selection is measured on its own.
Policy
500
1,000
2,000
4,000
8,000
recent
20%
21%
26%
32%
53%
relevant
33%
32%
42%
57%
71%
curated
35%
48%
68%
94%
100%
Share of the 100 questions whose correct answer is in the window with no stale value beside it. Columns are the budget in tokens.
The answer's presence in the window, by policy and budget. The two red lines are the keep-everything policy under two assumptions about the window.
Reading it:
Selection beats a sliding window at every budget. At 2,000 tokens a sliding window has the answer 26 percent of the time and curated 68 percent. The reason is simple: the answer is rarely in the last few steps. Questions in the last 20 steps ask about facts stated in the first 40.
Curation is what makes a small budget enough.relevant alone needs 8,000 tokens to reach 71 percent; curated is at 94 percent with 4,000. Supersede removes stale values, and trim removes the noise in each log, so more items fit.
"Keep everything" is not the ceiling. Even in a window big enough to hold all of it, keep_all has the correct answer without a stale value beside it in 76 percent of cases. In 24 of 100 questions the old value and the new one are both in the window. The answer is present; so is the wrong one, and which one a model picks is not under your control. (The modelled score is 0.64 for keep_all and 0.70 for curated at 4,000, on my invented curve; treat that as a direction.)
With a limit, "keep everything" is refused. With a 12,000-token limit, the request is refused from step 46, and 67 percent of the 100 questions get no answer at all. The 12,000 is imagined, but the shape is not: history grows by about 230 tokens per step and a limit is a fixed number.
The rule question is perfect for every policy (100 percent), because it lives in a pinned item. That is the point of pinning, and a planted defect that turns pinning off is caught by the first test that looks for the rule.
Past about 4,000 tokens curated stops improving, and its mean size drops to 6,219 at a budget of 8,000: it ran out of useful things to add. A bigger budget is not always filled. It is a limit, not a target.
The question behind this table
"Why not just put everything in a very long context?" Four honest reasons, each in this experiment: it is refused past the limit; it costs more at every step (section 8); corrections sit beside old values, so the answer is present and so is the wrong one; and long inputs are read less reliably (section 6). A good answer then says what you do instead: keep a store outside the window and build each window from it.
Where the lexical selector breaks
Split the same experiment by how the question was asked:
curated, budget
plain words
vague words
the rule
1,000
57%
12%
100%
2,000
82%
38%
100%
4,000
100%
85%
100%
A plain question ("What is the staging deploy key?") shares words with the line that holds the answer (deploy, key, staging), so word overlap finds it. A vague one ("Where does staging run?") shares only staging, a word on almost every line, so the selector cannot tell the right log from any other and mostly picks wrong. This is a real failure and I did not tune it away. It is the reason retrieval exists: module 4 replaces word overlap with search that understands meaning, and Level 4 (RAG) covered the mechanics. Here the lesson is only that the assembler is as good as the signal it ranks by.
Trimming has the same weakness in a smaller form. Trimming a log to 120 tokens never lost the answer line in all 80 of my trials (40 plain, 40 vague); trimming to 60 tokens lost it in 15 of 40 vague questions and in none of the 40 plain ones. So 120 sits just above a cliff, and I chose it by looking at this table. That is a choice of mine, flagged as one; your logs will have a different cliff.
Position and order
Needle in a haystack, lost in the middle
Needle in a haystack: a test that hides one fact (the needle) at different depths of a long input (the haystack) and asks the model to find it.
Where in the window a fact sits matters.
My reader has a curve with that shape, and it is an assumption: 97 percent at the best position, minus up to 30 points in the middle, minus 4 points for each doubling of length beyond 1,000 tokens. The shape below comes from my parameters, not from a model.
The same fact at 11 depths of an otherwise identical 4,000-token context. A U shape is what I put in. The experiment is what you rerun with a real model.
The assembler has one response to this: best_last, which puts the pinned rules first, keeps the rest in time order, and moves the single best-matching item to the end, next to the question. On my curve this moves the modelled score of curated at 2,000 tokens from 0.528 to 0.543, a small gain. It cannot change presence (the same items are chosen, tested), and it can hurt if your model reads the middle better than mine does or if the best-matching item is not the one the answer needs. Reordering is a bet; measure it (hard exercise 3).
Two other ordering rules are safer. Rules first and the question last is stable advice because both ends are where models read best. Keep the order stable between steps where you can: module 7 shows that a changing prefix defeats caching.