Part 5 of 6 · 4 min read
Exercises
Easy
Easy 1: Run and read (20 minutes)
In code/mini-query, run pytest, python pipeline.py and python demo.py. For each of the five transformations, find one question from questions.py where it was the right tool and one where it did not help. Use the baseline ranks to justify each.
Done when: you have ten short statements, each naming the words or missing words that decide the outcome.
Hint
use python pipeline.py for the table and the baseline ranks of the relevant passages to see why a question fails.
Easy 2: Rewrite by hand (15 minutes)
Rewrite these last messages so each stands alone, given the earlier message: (a) earlier "Which plan has audit logs?", last "and how much is it?"; (b) earlier "What does E-5503 mean?", last "does it happen often?"; (c) earlier "How do I install the SDK?", last "thanks, what about Node?". Say which of the three the example rewrite would get right and which not.
Done when: you have three rewrites and one sentence on a pattern the rule program cannot handle.
Hint
a pronoun or "that" points to the most recent thing the user asked about, but "what about Node?" replaces a part of the earlier question.
Easy 3: Which tool? (15 minutes)
For each question pick the transformation that fits best and say why: "my invoice looks wrong and also how do I cancel", "what's the fix for that?", "stuff is super slow lately", "where do I look up the delivery status of order 1182: help docs or the orders database?"
Done when: you gave one tool per question and a reason, and you noticed that one question might need two.
Hint
the third question is a vocabulary gap; the fourth is about where, not what.
Medium
Medium 1: Teach the rewriter (45 minutes)
Extend rewrite to handle two new follow-up patterns of your own (for example "and the price?" and "why?"). Write four new questions that use them, add them to questions.py, and add tests, including one that a standalone question is returned unchanged.
Done when: the tests pass and rewrite's follow-up score on the larger question set is reported before and after.
Hint
write the failing test first, then the pattern. Keep the "no history means no change" rule.
Medium 2: Keep the original (50 minutes)
Change HyDE so that it fuses the original query's list with the hypothetical's list using different weights (original counts double, equal, half). Report results by kind for each weighting.
Done when: you have a table of three weightings by kind and you can state which setting repairs the damage HyDE did to compound questions, or say that none does.
Hint
use weighted RRF (multiply each list's points) or repeat the original list.
Medium 3: A router that fails open (60 minutes)
Change route so it returns all routes whose vote count is within one of the best, and search the union of their subsets. Add tests, and compare compound-question results and the number of passages searched against the single-route version.
Done when: the tests pass, and you can explain what you gave up in exchange for fewer compound failures.
Hint
a union of subsets is a filter, so remember to keep the filter in both rankings, as hybrid does.
Hard
Hard 1: A real rewriter (120 minutes)
Replace the rule-based rewrite with a real model call using a prompt like "given the conversation, rewrite the last message so it stands alone; change nothing else". Build 20 conversations (10 follow-ups, 10 standalone) and measure how often the rewrite is correct, how often it changes a message that should not change, and what it does to retrieval. (I did not run this, so your numbers are the ground truth.)
Done when: you have a table of correct, unnecessary-change and harmful-change counts, and a conclusion on whether to skip rewriting for messages without pronouns.
Hint
tell the model to leave standalone questions unchanged, and count the cases where it did not.
Hard 2: Decompose with dependencies (120 minutes)
Write ten multi-hop questions where part two depends on part one (for example "what plan does the customer on invoice X have, and how long are its audit logs kept?"). Show why static decomposition fails on them, then write a small loop that retrieves for part one, reads the answer, forms part two and retrieves again, with a limit of three rounds. Compare with static decomposition.
Done when: you have a comparison table, a loop with a hard step limit, and a note on what the loop costs in calls (this previews module 8).
Hint
a loop needs a stop condition (found, or three rounds) so it cannot run forever.
Hard 3: Is it worth it? (90 minutes)
Make a cost model: assume a model call adds 600 ms, a search adds 40 ms and a call costs a fixed price of your choosing. For the seven systems, combine the example's calls and searches with its accuracy to compute added latency per question and cost per additional correct answer over baseline. Then state which system you would ship if the limit were 1 second and why.
Done when: you have a table with latency, cost and accuracy gain per system and a recommendation that names its assumptions.
Hint
cost per extra correct answer is total added cost divided by the number of questions fixed relative to baseline.