Part 4 of 5 · 4 min read
Exercises
Easy
Easy 1: Run and read (20 minutes)
In code/mini-agentic, run pytest and python demo.py. For the three-hop question in section 1 of the demo, list the three searches the agent made and, for each, what it learned. Then explain in one sentence why single pass returned "no answer".
Done when: you have three query and result pairs and a one-sentence explanation naming the missing word.
Hint
the first query's result gives the entity that goes into the second query.
Easy 2: Count the hops (15 minutes)
For each question, write the chain of facts needed and the number of hops: (a) "Which country is the capital of the company's biggest customer in?" (b) "Who wrote the manual for the product that replaced the model 3?" (c) "What is the refund policy?" (d) "Which libraries does the project that Maria leads depend on?"
Done when: you have four chains with hop counts, and you identify which one is plain single-pass material.
Hint
count the "of" and "that" links; each one is usually a hop.
Easy 3: Pick the system (15 minutes)
For each need, pick single pass, agentic loop or graph, with a reason: (a) a company FAQ that is edited daily; (b) "which suppliers are two steps away from supplier X" over a fixed supply network of 50,000 companies; (c) a helper that answers "what changed in policy last month and who approved it"; (d) a chatbot where 95% of questions are single lookups.
Done when: you have four picks, each naming one cost or risk of your choice.
Hint
edit frequency and corpus size matter more than how clever the system is.
Medium
Medium 1: A new relation (60 minutes)
Add a relation to the wiki (for example works_on: "Priya Nair works on the billing launch") and extend patterns.py, the planner and the query templates so a two-hop and a three-hop question using it work in all three systems. Add 4 questions and tests.
Done when: the tests pass and you report the accuracy of the three systems on the extended set.
Hint
add the regex first, then the template, then the question, then the test.
Medium 2: Fix the graph's blind spot (45 minutes)
Make the graph schema understand "reports to" and add two more phrasings of your own ("answers to", "is led by"). Re-run the graph accuracy. Then write a test that fails if a passage with a new phrasing is silently skipped, for example by counting passages that mention a known entity but gave no edge.
Done when: graph accuracy rises and your "silent skip" check flags a new phrasing you add.
Hint
look at docs_with_edges and docs_processed; they differ by exactly the skipped passages.
Medium 3: The loop's budget (60 minutes)
Add a cost model: each search costs 40 ms, each model call 600 ms. Compute average latency per question for the three systems. Then add a rule that sends one-hop questions to single pass and the rest to the loop, and report the accuracy and latency of this router.
Done when: you have a table of latency and accuracy for four systems and a recommendation.
Hint
a router needs a planner, so count its model call.
Hard
Hard 1: Real planner and reader (150 minutes)
Replace the planner and the reader with a real model. Give it 20 multi-hop questions over documents of your own (at least 30 passages, with distractors). Compare single pass and your agentic loop on accuracy, searches and calls, and record every case where the model wrote a bad query or misread a passage. (I did not run this, so your numbers are the ground truth.)
Done when: you have a results table, a list of failure cases with causes, and a step limit chosen from the data.
Hint
write the questions before you run anything, and include some that no passage answers.
Hard 2: Extraction audit (120 minutes)
Take 40 passages from documents you know. Write the triples a careful person would extract. Run a real model as an extractor on them and compute precision and recall of edges. Then measure how that affects answers to 10 multi-hop questions.
Done when: you report edge precision, edge recall, the question accuracy, and three examples of a wrong or missing edge with the effect on the answer.
Hint
count an edge as correct only if subject, relation and object all match.
Hard 3: Themes from the graph (120 minutes)
Add a community step: group nodes that are connected (for example by connected components), write a one-sentence summary for each group (by hand or with a model), and answer "What are the main areas of the company?" from the summaries. Show a question this answers that single pass and the loop cannot, and a question where the extra step makes things worse.
Done when: you have the grouping, the summaries, the two example questions, and a note on the cost of building and refreshing it.
Hint
connected components of this toy graph are chains; use a richer graph to see clusters.