Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-research.
Done when: you can explain why single retrieval answered 14 of 17.
Easy 2: Add a page (30 minutes)
Add a page with a new conflicting value and predict the output before running.
Done when: the conflict appears in the report as you predicted.
Hint
dates and kinds decide the winner; try a newer secondary page against an older primary one.
Easy 3: Read a report (30 minutes)
Print the statements for one question and annotate each with its source kind and date.
Done when: you can say which statement you would trust least and why.
Medium
Medium 1: Citation laundering (3 hours)
Add three secondary pages that all repeat one forum claim and an origin field.
Done when: the agent counts them as one origin and the report says so.
Hint
store the origin with each claim and group by origin before counting.
Medium 2: Better planner (3 hours)
Replace the keyword planner with a model call (or a stronger rule set) that handles questions in new wording.
Done when: you report plan accuracy on ten new questions.
Medium 3: Text-level verifier (4 hours)
Store claims as free text and verify by a quote or fuzzy match instead of a triple.
Done when: you report precision and recall of the verifier on fifty labelled statements.
Hint
record the exact quote used as evidence so a human can check it.
Hard
Hard 1: Real research eval (8 hours)
Evaluate a real model with search on twenty questions, scoring citation precision by opening every cited page. (I did not run this, so your numbers are the ground truth.)
Done when: you report coverage, correctness, citation precision and cost.
Hint
freeze the fetched pages so the evaluation can be replayed.
Hard 2: Cost and depth curve (6 hours)
Plot answer quality against pages read for your agent. (I did not run this.)
Done when: you identify the point beyond which extra reading did not help.
Hard 3: Adversarial pages (8 hours)
Write ten pages with planted instructions, fake authority and stale claims and measure what gets through. (I did not run this.)
Done when: you report which defences failed and add one.