Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-traces.
Done when: you report coverage of the 1 percent category for the three strategies.
Easy 2: Chance of coverage (30 minutes)
Compute the chance a random set of n cases contains at least one 1 percent case.
Done when: you give n for 95 percent and compare 50 cases with the simulation (39 percent).
Hint
1 minus 0.99 to the power n.
Easy 3: Weight by hand (30 minutes)
Using the traffic shares, compute the weighted failure rate from per-category rates of 5, 10, 8, 40 and 55 percent.
Done when: it matches weighted_estimate logic on a sample with those rates.
Hint
sum of share times rate.
Medium
Medium 1: Inclusion weights (3 hours)
Fix the flagged estimator with inverse-probability weights.
Done when: the bias falls below one point and you explain why category weights failed.
Hint
weight each sampled case by the inverse of its selection probability.
Medium 2: Better deduplication (3 hours)
Normalise numbers and ids before comparing, and add an embedding-style similarity (for example character n-grams).
Done when: the 20 order-number questions collapse to one and you list one false merge.
Hint
replace digits with a token before computing shingles.
Medium 3: Slice sizes (2 hours)
Compute the interval half-width for slices of 10, 30 and 100 cases at a 90 percent pass rate.
Done when: you give the smallest slice you would report.
Hint
reuse flaky.wilson from module 1.
Hard
Hard 1: Real traces (6 hours)
Build a 100-case set from real traces with scrubbing, deduplication and two annotators. (I did not run this, so your numbers are the ground truth.)
Done when: you report agreement, slice sizes and what you removed.
Hint
scrub before anyone labels.
Hard 2: Leakage audit (4 hours)
Check an existing prompt and eval set for overlap. (I did not run this.)
Done when: you list leaked items and the score change when they are removed.
Hint
use traces.leaked.
Hard 3: Drift (5 hours)
Compare a set from last month with this month's traffic. (I did not run this.)
Done when: you report which slices changed share and how the score would shift under weights.
Hint
compare category shares, not just scores.