Part 5 of 6 · 4 min read
Exercises
Easy
Easy 1: Run it and read the table (30 minutes)
In code/mini-memory, run pytest -q (about two seconds), then python demo.py and python run_all.py.
Done when: you report, for typed, the recall by question type from results.json, say which type failed and why in one sentence, and say what transcript did on that type at a 6,000-token budget.
Hint
evaluate.run("typed")["by_cat"] gives the split; evaluate.transcript_sweep() gives the replay budgets.
Easy 2: Teach the extractor one more fact (40 minutes)
Add a pattern for "My timezone is X" to extract.py (key timezone). Add three tests: it matches, a paraphrase does not, and a tool result with the same sentence is quarantined.
Done when: the three tests pass, python mutants.py still reports the same three escapes, and you say whether adding the pattern changed any experiment number (it should not).
Hint
add the pattern to FACT_PATTERNS; the test for the tool case calls extract.apply(store, extract.propose(text), "tool", 1, "gated").
Easy 3: Look inside the file (40 minutes)
Build a world to a file (memory.build(world.make_world(0), "gated", "mem.db")) and open it with the sqlite3 command or Python's sqlite3. List the four tables and their rows. Then repeat after until=7 (before the forget) and until=12.
Done when: you show the rows for phone before and after, find the row in quarantine, and say which table would have kept the phone number if scrub were off and the summary had quoted it.
Hint
PRAGMA table_info(facts) lists columns; SELECT * FROM facts shows the current flag.
Medium
Medium 1: Close the three holes (2 hours)
Write one new test for each escape in section 11 (a default-gate test, an assistant-line test, an ordering-independent procedure test) and fix store.py or extract.py if needed. Re-run python mutants.py.
Done when: all 28 mutants are caught, you show which new test catches each one, and you say which of the three you would have found without the mutation run.
Hint
the default-gate test calls apply(store, props, "tool", 1) with no gate; the assistant test calls memory.build on a world whose assistant lines contain a pattern; for procedures check SELECT COUNT(*) ... WHERE current = 1.
Medium 2: Facts that expire (3 hours)
Add an optional expiry to facts: "This week I am in Berlin" should become a fact location that stops being returned after N sessions. Extend the extractor, the store and recall, and add a world event.
Done when: a question asked inside the window returns Berlin and one asked later does not, an expired fact is deleted rather than hidden (check the file bytes), and a planted defect (expiry ignored) is caught.
Hint
store the expiry session in a new column; filter in facts(); delete expired rows at the start of each session.
Medium 3: Review the quarantine (3 hours)
Write review(store, id, accept): accepting a quarantined proposal writes it (with source reviewed), rejecting deletes it. Add a world event where a useful fact arrives through a tool (a calendar says "timezone: Asia/Kolkata").
Done when: the poisoned proposal can be rejected and the useful one accepted, the audit trail shows who accepted what, and you explain what stops a tired reviewer from accepting the poisoned one (it is a design question; there is no code answer).
Hint
reuse set_fact with a new source name and add it to TRUSTED only inside review, not globally.
Hard
Hard 1: A real extractor (4 hours)
Replace the patterns with a call to a real model that proposes memory writes from a turn. Write 30 statements (10 plain, 10 paraphrases, 10 that should not be remembered, such as jokes and hypotheticals). (I did not run this, so your numbers are the ground truth.)
Done when: you report recall on the first two groups and false writes on the third, the model's name and the date, and say which of my choices (strict patterns, the gate, the set of keys) you had to change.
Hint
score by comparing the key and the value; for the "should not remember" group, count every write as a false write.
Hard 2: Your own sessions (4 hours)
Take 10 real conversations with an assistant (yours, or invented from real tasks) and 20 questions whose answers are in them. Run none, transcript, flat, semantic and typed on them. (I did not run this.)
Done when: you report recall by policy, say where the fact table was too small (what did not fit the three kinds?), and say what you would add as a fourth kind of memory.
Hint
keep the answers as strings so evaluate.judge works unchanged.
Hard 3: Attack the gate (5 hours)
Find three ways to get an attacker's sentence into memory despite the gate: for example, text the user pastes, an assistant summary of a tool result that is then confirmed by the user, or a fact that looks like a procedure. Write each as a world event and run it. (I did not run this.)
Done when: you report which attacks worked, fix the code for at least one, add a test and a planted defect, and say which attack the gate cannot stop at all.
Hint
remember that the gate looks at the source label, so the attack is anything that makes the label wrong.