Part 1 of 4 · 1 min read
Overview
Without this, teams ship a prompt change because the score went from 0.72 to 0.75 on 20 tasks. Interviewers ask "how do you test an agent whose output changes on every run?", and the answer is statistical: repeated trials, intervals, enough tasks, and consistency metrics.
About the example, and what is real
Real: the Wilson interval, pass@k and pass^k formulas, the sample-size formula, and the seeded Monte Carlo in code/mini-flaky, with 11 tests. Assumed: task difficulty is Beta-distributed (mean 0.7 or 0.75, so some tasks are easy and some hard); each try is independent given the task; per-step reliabilities in the chain table; a perfect detector for retries. No model is called. The numbers show how noisy scores can be, not how noisy yours are; measure that with your own repeated runs.