Part 4 of 5 · 4 min read
Exercises
Easy
Easy 1: By hand (20 minutes)
A system returns [d3, d9, d1, d7, d5]. The relevant set is {d1, d3, d8}. Compute precision@3, recall@3, hit@1, reciprocal rank, and average precision. Then swap the order of d9 and d1 and recompute each.
Done when: you have ten numbers and a sentence on which metrics changed and which did not.
Hint
AP divides by the number of relevant passages (3), including the one never retrieved.
Easy 2: Run and read (20 minutes)
In code/mini-metrics, run pytest and python demo.py. Find a query where one system has recall@3 of 1.0 but RR of 0.5. Say what that means for a model that reads the top 3 and for a person who clicks the first result.
Done when: you name the query and give both interpretations.
Hint
recall@3 equal to 1.0 with RR of 0.5 means every relevant passage is within three, but none is first.
Easy 3: Pick the metric (15 minutes)
For each system, name the main metric and why: (a) a FAQ bot showing the single best answer; (b) an agent that puts 5 passages in a prompt; (c) a document search page showing 10 results with "most relevant" and "somewhat relevant" labels; (d) a legal search where missing any relevant case is costly.
Done when: you have four metric choices with a reason each, and (d) mentions a recall-oriented metric with a large k.
Hint
(c) has two relevance levels, so one metric uses them directly.
Medium
Medium 1: Build graded labels (60 minutes)
Write 12 new queries for the example help centre, including two that need two passages and two with no answer. Label each passage as 2, 1 or missing and add answer phrases. Add them to evalset.py and re-run the table.
Done when: the tests still pass, and you report how each system's recall@3 and MRR moved when the set grew from 24 to 36 queries.
Hint
write the query first, then decide the labels without looking at any ranking.
Medium 2: Intervals and sample size (60 minutes)
Using the bootstrap helpers, show how the width of the 95% interval for MRR changes when you keep only the first 6, 12 and 22 queries. Then estimate how many queries you would need for the interval to be about half as wide as it is at 22.
Done when: you have three interval widths and an estimate with your reasoning (hint: widths shrink with the square root of the sample size).
Hint
keep the same seed; compare hi - lo.
Medium 3: Detect pooling bias (45 minutes)
Run judged_only for each of the four systems as the labelled one. Report the gap between each system's recall@3 under its own labels and under complete labels. Design a labelling protocol (which passages go to the annotators) that keeps the gap under 0.03 in this example.
Done when: you have a table of gaps and a protocol, tested by simulation.
Hint
a protocol that unions the top results of several systems removes most of the bias.
Hard
Hard 1: Real documents, real labels (150 minutes)
Take 40 or more passages from documents you know (module 1's pipeline) and write 30 real-sounding queries, including 5 with no answer. Label them with grades. Compare a keyword retriever, a real embedding retriever and hybrid fusion on recall@3, MRR and nDCG@3, with paired bootstrap intervals. (I did not run this, so your numbers are the ground truth.)
Done when: you have a table, intervals, and a written verdict that is allowed to say "cannot tell".
Hint
freeze the labels before you look at any system's output.
Hard 2: Judge the labels (120 minutes)
Have a language model label every (query, passage) pair for 20 of your queries (grades 0 to 2). Compare its labels with yours: agreement, and the effect on each system's recall@3 and MRR when you use its labels instead of yours.
Done when: you report agreement and whether the ranking of systems changed, plus two disagreements you looked at.
Hint
compare the order of systems, not just the absolute values.
Hard 3: From retrieval to answer (120 minutes)
For 30 queries run the whole pipeline (retrieve top 3, answer with citations as in module 5). Record recall@3 and whether the final answer was correct. Estimate how often a miss in retrieval leads to a wrong answer and how often a hit still leads to a wrong one.
Done when: you have a 2 x 2 table of retrieval hit or miss against answer right or wrong, and one sentence on where improvement would pay off most.
Hint
a retrieval hit that still gives a wrong answer points at module 5 or 6, a miss at modules 1 to 4.