Covers: Definitions; The pipeline; Results; Design rules; Interview questions, with the follow-ups; Common mistakes; What the example does not do.
Definitions
Trace, eval case, golden set
Trace: the recorded input, steps and output of one agent run.
Eval case: an input plus what counts as success (an expected answer, a check, or a rubric).
Golden set: a curated, labelled set of cases treated as ground truth.
Slice, stratified sample, post-stratification
Slice: a subset of cases defined by an attribute, such as topic or user tier.
Stratified sample: a sample with a fixed number from each slice.
Post-stratification (weighting): correcting an estimate by weighting each slice by its real share of traffic.
Near-duplicate, Jaccard similarity, leakage
Near-duplicate: two items that differ only trivially.
Jaccard similarity: the size of the overlap of two sets divided by the size of their union; here, of their word-trigram sets.
Leakage: an eval item appearing in the prompt, the examples or the training data.
The pipeline
Collect, scrub, dedupe, sample, label and freeze.
Results
Coverage of a 50-case set (2,000 draws from 20,000 traces):
Strategy
Contains the 4% category
Contains the 1% category
Random
88%
39%
Flagged-weighted (half flagged)
99%
70%
Stratified (10 per category)
100%
100%
Estimate of the overall failure rate (true 8.8%; 600 draws of 50 cases):
Strategy and estimator
Bias (points)
RMSE (points)
Random, naive
+0.2
3.9
Random, weighted
-0.3
3.8
Stratified, naive
+18.8
19.6
Stratified, weighted
+0.2
4.6
Flagged, naive
+21.0
21.7
Flagged, weighted
+16.5
17.4
Coverage of the rare category and error of the estimated failure rate for three sampling strategies.
Deduplication: 43 items (20 templated order questions, 20 case-and-punctuation variants of them, 3 distinct) became 23 at a similarity threshold of 0.5 and also at 0.9. The 20 questions differing only in an order number were kept, which is the limitation to fix with normalisation. Leak check: an item present in a few-shot example was flagged.
Choose the sample for the question. Overall quality: random, large, weighted. Finding failures and covering rare behaviours: stratified or flagged, read per slice.
Never read a stratified or flagged set's overall score as production quality without weights, and flagged selection needs inclusion-probability weights.
50 random cases miss a 1 percent category more often than not; if it matters (injection, a safety case), add it deliberately.
Deduplicate before counting, and decide what "the same" means for your task.
Design rules
Version the set and record where each case came from (date, trace id, annotator).
Slice and report per slice, with intervals (module 1): slices of 10 cases have wide error bars.
Add every production failure you fix as a regression case (module 7).
Refresh on a schedule, because traffic drifts; keep the old set for trend comparison.
Label with a written guideline, have two people label a subset and measure agreement (module 4).
Scrub, then store: remove personal data and secrets before traces become test data.
Keep a hold-out that nobody tunes against.
Interview questions, with the follow-ups
How do you build an eval set for an agent that is already live?
"From production traces. I collect them, scrub personal data, deduplicate, and sample in two ways: a random, production-shaped set to estimate overall quality, and a stratified set with deliberate coverage of slices and rare failures. I label against a written guideline, check agreement on a subset, version the set and freeze it. In my simulation a random 50-case set contained a 1 percent category 39 percent of the time and a stratified one every time." Follow-up: "How do you get an overall number from the stratified set?" Weight each slice by its traffic share; naive averaging over-stated failures by 18.8 points in my test.
Why not just use the thumbs-down traces?
"They are biased: users flag some failures and not others, and flag some passes. In my simulation, a set built on flagged traces over-stated the failure rate by 21 points, and slice weights did not remove the bias because selection depended on the outcome. They are good for finding failure types, so I use them for the failure set and use a random set to estimate the rate." Follow-up: "Can you correct it?" Only with known inclusion probabilities, or by labelling a random sample.
What can go wrong with an eval set?
"Duplicates that inflate one task, leakage into the prompt or few-shot examples so the score measures memory, stale cases as traffic drifts, tiny slices with huge error bars, and labels that disagree. I deduplicate, check overlap with prompts, refresh, report intervals, and measure annotator agreement." Follow-up: "How do you know the set is big enough?" The interval must be narrower than the change I want to detect (module 1), per slice that I care about.
Can a model write your eval cases?
"Yes, to extend a real set: mutate real cases, add ambiguity, produce edge cases. Humans must review them, and I keep them labelled as synthetic and out of the production-shaped set, because they do not have the real distribution and can share the generating model's blind spots." Follow-up: "What is the risk?" Evaluating a system with cases from the same model family can overstate quality.
Common mistakes
Tuning prompts on the eval set and reporting its score.
Few-shot examples copied from the eval set.
Reading an oversampled set as the production rate.
No slices, so a regression in a small, important group is invisible.
Counting duplicates as independent cases.
Unlabelled provenance, so nobody knows what the set represents.
Personal data in test cases.
Never refreshing.
What the example does not do
No real traces or annotators; the failure label is known exactly.
Five categories only, assigned by construction; real slicing is messy.
Weighting by category only; real corrections can need several variables.
Simple lexical deduplication: no embeddings, and no normalisation of numbers or names.
No privacy scrubbing (Level 10 module 6).
Optional real experiment (I did not run this): take 500 real traces, label 100 random and 100 stratified, and compare the estimated rate and the number of distinct failure types found.
Check yourself
You sample 50 random traces and one category is 1 percent of traffic. Will the sample tell you about it? (Usually not: that category appeared in the sample only 39 percent of the time in the simulation. Stratified sampling always had it, but then each category must be weighted by traffic share to estimate the overall rate.)
For the curious Stratified sets find failures but misread the rate
True failure rate was 8.8 percent. A random sample was off by 0.2 points. A stratified sample read naively was off by 18.8 points, because rare, failure-prone cases were over-represented; weighting each category by its traffic share brought it back to 0.2. Keep two sets: a production-shaped one to estimate quality and a stratified one to find and guard failures. All traffic shares and rates are assumptions.
What you will use daily at work
Practice
What it looks like
Version every eval set
Record where each case came from: date, trace id, annotator.
Report per slice with intervals
A slice of 10 cases has very wide error bars.
Turn each fixed production failure into a case
It then runs in CI as a regression test.
Scrub personal data first
Remove it before logs become test data.
Draft: confirm how your tracing tool exports traces and which fields need redaction.
What you must be able to say (one page)
Production traces are the best source of eval cases, because they show what users actually ask. Curate annotated examples for each component and criterion, use actual production data where you can, slice the data (by user tier, traffic source, topic) and keep several evaluation sets, with one that mirrors the real distribution to estimate overall quality. Cases can be added by hand, mined from logs, or generated by a model and then reviewed by humans.
The sampling method decides what the set can tell you. In my simulation (five categories; the rarest, 1 percent of traffic, has a 60 percent failure rate), a random 50-case sample contained that category only 39 percent of the time; sampling with a weak "flagged" signal 70 percent; stratified sampling 100 percent.
A set that covers rare cases no longer estimates the overall rate unless you weight it. True failure rate 8.8 percent. Random sample: error +0.2 points. Stratified sample read naively: +18.8 points (rare, failure-prone cases are over-represented). Stratified with each category weighted by its traffic share: +0.2 points, at an RMSE of 4.6 points against 3.9 for random. Sampling on the flagged signal stays biased even with category weights (+16.5) because selection depends on the outcome; you need inclusion-probability weights or a separate random set.
Two sets, two jobs: a random, production-shaped set to estimate quality, and a stratified or adversarial set to find failures and protect rare behaviours.
Deduplicate before you count. Near-copies (case and punctuation changes) inflate the suite and overweight one task; 43 items shrank to 23 with word-trigram Jaccard similarity. Numbers differing inside a template are not caught by this measure; normalise them first.
Keep the set out of your prompts. If an eval item (nearly) appears in few-shot examples, the score measures memory. Contamination is a known problem for public benchmarks, and the same applies to your own set.
Use synthetic cases to extend, not replace: a model can mutate real cases into edge cases, and humans review them.
Scrub personal data before logs become test data (Level 10, data privacy).
Honest limits: traffic shares, failure rates and the flag's accuracy are my assumptions.