Part 1 of 5 · 1 min read
Overview
In this module you will
- Compute precision@k, recall@k, hit rate, reciprocal rank, average precision and nDCG by hand and in code.
- Pick the metric that matches how your system uses the results.
- Build an evaluation set with queries, graded relevance labels and questions that have no answer.
- Put an interval around a score and tell a real improvement from noise.
- Name three ways a test set can mislead you: too few queries, incomplete labels, and labels made from one system's output.
Every earlier module in this level made a claim such as "hybrid beats keyword". Without numbers you cannot tell whether a change helped, hurt, or only changed the examples you happened to try. Retrieval metrics are also the cheapest, fastest part of evaluating a RAG system: they need no language model call, run in milliseconds, and point at where a failure began. If the right passage never reaches the model, no prompt will fix the answer. This module teaches you to measure that stage, and to be honest about how much a number can be trusted.
About the examples
The corpus is the same invented 18-passage help centre used in module 3, with a new evaluation set: 24 queries, of which 22 have labelled answers (graded 2 = answers the question, 1 = useful related context) and 2 have no answer in the documents. I wrote the labels by hand. The four retrievers are the module 3 imitations (keyword, a toy vector search, their fusion, and a toy reranker). With 22 scored queries one query moves a score by about four points. Treat the figures as a demonstration of the method and its pitfalls, not as benchmarks. No real model or service was called.