Part 4 of 5 · 4 min read
Exercises
Easy
Easy 1: Run and read (20 minutes)
In code/mini-hybrid, run python evaluate.py, python demo.py and pytest. In the per-query table from demo.py, find one question where keyword search wins, one where vector search wins, and one where the hybrid is worse than the better single method. Explain each in one sentence.
Done when: you have three questions with three explanations that name the words (or missing words) responsible.
Easy 2: RRF by hand (15 minutes)
List A ranks passages p, q, r, s. List B ranks r, p, t, q. With k = 60, compute the RRF score of each passage and give the final order. Then recompute with k = 1 and say what changed.
Done when: you have two orderings, and you can explain what a small k does to first place.
Hint
with k = 1 first place is worth 1/2 and second 1/3, so the gap between ranks is much bigger.
Easy 3: Which retriever wins? (20 minutes)
For ten questions you might ask a help centre (an error code, "how do I get my money back", a product name, "why is it slow", a version number, a person's name, and so on), predict whether keyword, vector or hybrid is safest, and why. Then check two of them with the example by adding passages.
Done when: you have ten predictions and two checks, and you noticed at least one prediction that was wrong or too confident.
Hint
think about which words in each question would also appear in the right passage.
Medium
Medium 1: Tune BM25 (45 minutes)
Sweep k1 in 0.5, 1.2, 2.0 and b in 0, 0.5, 1.0. Add four long passages (10 or more sentences) and four short ones to the corpus, plus questions that should match them. Report top-1 and MRR for the nine settings.
Done when: you have a nine-row table and one sentence on what b = 0 and b = 1 do to long passages.
Hint
b controls how much passage length is penalised; k1 controls how fast repeated terms stop helping.
Medium 2: Fusion shootout (50 minutes)
Compare RRF with k in 1, 10, 60 and 200 against weighted fusion with alpha in 0, 0.25, 0.5, 0.75, 1 on all 18 questions, by kind. Add six more questions of your own that are hard for both retrievers.
Done when: you have a table and a recommendation, and you can say which method is more sensitive to its setting.
Hint
for weighted fusion, check what happens when all scores in one list are equal.
Medium 3: Filter before or after (60 minutes)
Give each passage a plan tag (starter, zephyr, or all). Write search(query, plan, k, mode) where mode pre removes other plans before ranking and mode post ranks everything and then removes. Show a question where post returns fewer than k results and pre does not. Test both modes.
Done when: the tests pass and you can explain why this matters more for approximate vector indexes than for this small example.
Hint
with post, take the top k after retrieval, then filter, and compare the count returned.
Hard
Hard 1: Real components (120 minutes)
Take a set of real documents you know (at least 30 chunks from module 1's pipeline) and write 24 questions, eight of each kind (exact term, paraphrase, mixed). Compare keyword search (your own BM25 or a library), a real embedding model, RRF and (if you have access to one) a real reranker. Report top-1, recall at 5 and MRR by kind, and the extra latency of the reranker. (I did not run this, so your numbers are the ground truth.)
Done when: you have a table by kind, and a written verdict that is allowed to say a component did not help on your data.
Hint
write the questions before looking at any retrieval results, and keep the exact-term questions truly exact (real codes and names from your documents).
Hard 2: Learn the reranker (120 minutes)
Replace the hand-made reranker score with a small linear model over four features: coverage, proximity, BM25 score and dense similarity. Train it (logistic regression or a simple perceptron on correct versus incorrect passages) using 12 of the 18 questions and test on the other 6. Repeat with three different splits.
Done when: you report test top-1 for the three splits next to the hand-made reranker, and you state honestly whether 6 test questions are enough to trust the difference.
Hint
standardise the features (subtract the mean, divide by the spread) before training, and keep the split fixed while you debug.
Hard 3: The shortlist sweep (90 minutes)
For shortlist sizes N in 1, 2, 3, 5, 8 and 18, measure (a) first-stage recall at N, (b) top-1 after reranking, (c) pairs scored per query. Plot or tabulate the three against N, find the smallest N within 2 points of the best top-1, and explain why top-1 can never exceed recall at N.
Done when: you have a six-row table, a chosen N with a reason, and the one-sentence argument about the ceiling.
Hint
recall at N is "is the right passage in the first N of the fused list"; reranking cannot change that set.