Part 1 of 4 · 1 min read
Overview
A prompt tweak that fixes one customer complaint quietly breaks five other behaviours. Without a gate you learn that from users. With a bad gate (noisy threshold) engineers learn to ignore the red build. The interview question is "how do you stop a prompt change from degrading the agent?", and the answer is a statistically honest, paired, multi-trial gate with critical cases.
About the example, and what is real
Real: the exact paired sign test, the gate logic, the command-line script and its exit codes, flaky-case detection and the seeded simulations in code/mini-gate, with 11 tests. Assumed: per-case pass rates are Beta-distributed with mean 0.75; a "regression" lowers every case's pass rate by the same amount; cases are independent. Not run: the GitHub Actions file in ci/ is an example only. No model is called. The simulation shows the trade-off between false failures and missed regressions; it does not tell you the right setting for your suite.