Part 5 of 6 · 5 min read
Exercises
Easy
Easy 1: Run it and read the table (30 minutes)
In code/mini-window, run pytest -q -x (about a minute), then python demo.py and python run_all.py. Open results.json.
Done when: you report, for recent, relevant and curated at a budget of 2,000, the share of answers present, and explain in two sentences why recent is lowest.
Hint
evaluate.run_policy(policy, budget, limit=10**9) gives one row. The limit argument lifts the imagined window limit.
Easy 2: Count the wrapper (30 minutes)
Using assemble.cost and Item.tokens, find the wrapper overhead for the system item and for a one-line user message. Change the wrapper in fence (for example, shorter tags such as <<t id=...>>) and re-run evaluate.cost_of_wrapper().
Done when: you report the old and new overhead per item, say how many more items fit at a budget of 2,000, and say what you gave up by shortening the tags.
Hint
the wrapper is fence; its cost is cost(item) - item.tokens. The 15-token figure is an average.
Easy 3: Forge it yourself (40 minutes)
Write three different forged strings that you think could get past escape (for example, using different bracket characters, or splitting << across two items). Run each through fence and parse_roles.
Done when: you show each attempt and its result, say whether any succeeded, and say what kind of attack the fence does not address at all.
Hint
the escape works on the two-character sequences << and >>. Think about what happens at the boundary between an item and the wrapper text around it.
Medium
Medium 1: Make the question vague-proof (2 hours)
In assemble.py, make relevant and curated work better on vague questions without any model: for example, expand the query with a small synonym table you write by hand (run to region, copies to replicas), or boost items from the environment named in the question.
Done when: you report plain and vague presence for curated at 1,000, 2,000 and 4,000 tokens before and after, a new test that fails without your change, and one example where your table makes things worse.
Hint
put the expansion in words() or in score(); keep it out of the stored items so the answer lines are unchanged.
Medium 2: A second kind of staleness (3 hours)
Facts here are replaced by a later item with the same key. Add a different kind: a deletion ("config removed: staging.region"). Extend the world so some questions ask about removed facts (the right answer is "not set") and extend latest_only to handle it.
Done when: your assembler answers correctly, a new planted defect (ignoring deletions) is caught by a new test, and you explain what a stale answer to such a question would cost in a real system.
Hint
add a new Item field or a convention in the text (removed:), and a question kind with answer not set.
Medium 3: Choose the budget (2 hours)
Run curated at budgets of 1,000 to 8,000 in steps of 500. Find the smallest budget at which presence reaches 90 percent, and the smallest at which it reaches 100 percent. Add the total tokens sent over the run for each.
Done when: you give both budgets, the tokens-per-run cost of each, and a one-paragraph recommendation that says what you would pay for the last ten percentage points.
Hint
evaluate.run_policy returns mean_tokens; multiply by the number of steps for the run total.
Hard
Hard 1: Your own logs (4 hours)
Replace the generated world with real text: for example, a thousand lines of your own build or application logs with five facts you know are in them, and fifteen questions you write. Run all four policies. (I did not run this, so your numbers are the ground truth.)
Done when: you report presence by policy, say where the selector did well and badly, say which of my choices (best-line scoring, the 120-token trim, the idf weighting) you had to change, and why.
Hint
keep ground truth in a small file so you can score by string match, as reader.presence does.
Hard 2: A real model reads the context (4 hours)
Send the contexts that curated and recent produce for 30 questions to a real model, with the same instruction. Record whether it answered correctly, and whether the answer was present. (I did not run this.)
Done when: you report accuracy given presence, and accuracy given absence, with the model's name and the date, and say whether a model ever answered correctly from a context where the answer was absent (and what that means for my "presence" check).
Hint
use the same system rule in both runs, and do not tell the model which policy produced the context.
Hard 3: Measure the position curve (5 hours)
Reproduce the needle-in-a-haystack experiment of section 6 with a real model: the same fact at 11 depths in a 4,000-token context, 20 questions per depth, and the same again at 16,000 tokens. Replace reader.PARAMS with your fitted values and rerun best_last_gain. (I did not run this.)
Done when: you plot both curves, state whether the middle is worse for your model, say whether best_last helped or hurt with your curve, and say how far your numbers are from mine.
Hint
put the needle in at exact token offsets by building the filler first and inserting the fact line at the chosen index.