Trusting a number

The measured picture

At k = 3 over the 22 answerable queries:

No column ranks the systems in a different order (some columns tie), which is reassuring. But the choice of k and metric can flip a comparison. At k = 3 keyword beats vector on recall (0.86 against 0.77). At k = 5 vector is ahead (0.89 against 0.86) while keyword still leads on MRR (0.86 against 0.77): vector search retrieves more of the relevant passages somewhere in its top five but puts them lower. A statement like "A is better than B" is incomplete without the metric and the k.

How much can you trust a number?

Twenty-two queries is a small sample. Resample the queries with replacement 2,000 times (a bootstrap) and see how far the score moves:

The intervals overlap heavily. A better test compares two systems on the same queries (a paired comparison), which removes the variation that comes from some queries being easy for everyone:

Every interval includes (or just touches) zero. The honest summary is: the reranked system looks best, but 22 queries cannot confirm it. To tell a four-point gain from luck you need many more queries, or a gap much larger than four points. Report intervals, and treat a change inside the interval as "no evidence yet".

Building a test set that does not lie

1. Collect real queries from logs, support tickets or your own use, and keep their wording (typos included). Invented, tidy queries flatter every system. Use generated questions to expand coverage, not to replace real ones.
2. Label with grades, not just yes or no ("answers it", "useful context"). You will need them for nDCG, and they force you to decide what "relevant" means.
3. Include questions that have no answer, and decide what the right behaviour is (return nothing, or return passages that a later step will refuse). The example has two. For both, the keyword scores are 2.77 and 1.48, while real answerable queries score from 0 to 9.5: no score threshold separates them cleanly, which is why the gate in module 5 looks at the question's words, not just a retrieval score.
4. Slice by kind of question (exact term, paraphrase, multi-part, follow-up), as module 3 did. A good average hides a weak group.
5. Watch for pooling bias. If people labelled only the passages that one system returned, passages that other systems find are marked "not relevant" by default, and the labelled system looks best.

Judging only one system's results gives that system a perfect recall by construction. Label from the union of several systems' top results, plus some random passages as a control.

6. Count information, not only passages. A passage-level recall of 0.50 can still carry the whole answer. In the example, for three queries, label recall is 0.50 but the retrieved text contains every required answer phrase (fact recall 1.00), because the missed passage was only "related context". Judge-based context recall, as in common RAG evaluation toolkits, works at this level: does the retrieved text support each claim in the reference answer?

Practical advice

1. Measure retrieval before generation. If recall@k is low, fix retrieval first (modules 1 to 4).
2. Fix a test set and a metric before you change things. Re-run after every change.
3. Use the same k the model really sees.
4. Always show an interval or at least the number of queries.
5. Keep a failures list. For each query where relevant passages were missed, write down why (chunking, vocabulary, ranking). That list guides the next module's work.
6. Combine with an end-to-end check. Good retrieval metrics do not guarantee good answers (modules 5 and 6; full pipelines in Level 9).
7. Refresh the set as documents and users change.

For the curious Undefined cases now raise an error

recall_at_k and average_precision divided by zero for a query with no relevant passages, and precision_at_k did the same for k = 0. They now raise a ValueError that says what to do (leave unanswerable queries out of recall and MAP, and judge them separately by whether the system refuses). The bootstrap functions also refuse an empty list and unequal lists, because a paired comparison on different queries means nothing.