Covers: Definitions; The pipeline; Results; Design rules; Interview questions, with the follow-ups; Common mistakes; What the example does not do.
Definitions
Regression, baseline, quality gate
Regression: a change that makes behaviour worse than before.
Baseline: the stored results of the currently deployed version on the eval set.
Quality gate: an automatic check that passes or fails a change.
Paired comparison, sign test, McNemar
Paired comparison: comparing two versions on the same cases and using each case's difference.
Sign test: counts cases that got better (wins) and worse (losses) and tests whether wins and losses are balanced; with pass or fail outcomes this is the exact form of McNemar's test.
False failure: the gate fails a change that did not make anything worse.
Critical case, flaky case, quarantine
Critical case: a case whose failure is unacceptable regardless of the average.
Flaky case: a case with an unstable result.
Quarantine: keeping a case in the suite but not letting it fail the build.
The pipeline
A pull request triggers the suite, a paired comparison with the baseline, the gate rules and a report.
Results
Share of builds that fail (600 repeats; 0 means the candidate is identical to the baseline):
Suite
Naive gate, nothing changed
Paired gate, nothing changed
Naive gate, true 5-point drop
Paired gate, true 5-point drop
30 cases, 1 run
45%
2%
65%
7%
30 cases, 3 runs
37%
2%
72%
17%
100 cases, 1 run
30%
3%
68%
20%
100 cases, 3 runs
23%
4%
82%
45%
300 cases, 1 run
23%
5%
82%
48%
300 cases, 3 runs
9%
3%
94%
88%
False failures and detection of a true 5-point drop for a naive and a paired gate, in three suite sizes.
The naive gate fails identical builds because suite noise (module 1) is larger than its tolerance.
The paired gate is conservative: it nearly never blocks a good change, and needs a large suite to catch small drops.
Three runs per case is cheaper than 3 times the cases for a noisy model, and in this simulation raised detection from 20 to 45 percent at 100 cases; but more cases did more (48 percent at 300 cases, one run).
A critical case needs a rate of 1.0, which means more trials for those cases, or a deterministic check.
Design rules
Version and store results: case id, pass rate, trace link, model and prompt version, configuration and date.
Gate in layers: critical cases (no tolerance), a paired statistical test for the average and each important slice, and cost and latency limits.
Pin what you can: model version, temperature, tool versions, recorded tool outputs.
Use several trials and compare per-case pass rates.
Quarantine flaky cases, track them, and fix them within a deadline.
Make failures actionable: list the cases that got worse, with links to old and new traces.
Promote the baseline on merge, not on every commit, and keep old baselines for trend charts.
Keep a fast smoke set for every commit and the full suite nightly or on release branches.
Add every production incident as a new critical case (module 2).
Rerun everything when a tool changes, not only the cases that mention it.
Interview questions, with the follow-ups
How do you stop a prompt change from degrading your agent?
"A gate in CI that runs the eval suite on every change and compares per case against the stored baseline. It fails on critical cases, on a statistically supported drop, or on a cost limit. In my simulation a naive threshold gate failed 45 percent of identical builds with 30 cases, whereas a paired sign test failed 2 percent; the cost was power, 7 percent detection of a 5-point drop with 30 cases, 88 percent with 300 cases and three runs each." Follow-up: "What do you do when the build is red?" The report lists the cases that got worse with traces, and I either fix the change, or accept it with a recorded justification and update the baseline.
Why not simply require the score not to fall?
"Because the score is a noisy measurement. Run-to-run variation alone makes a bare threshold fail identical builds, and engineers learn to rerun until green, which defeats the gate. The paired test uses per-case wins and losses, which removes the shared case difficulty and tests whether more cases got worse than better." Follow-up: "Isn't that too lenient?" Yes for small drops on small suites; that is why critical cases have zero tolerance and why I increase cases and trials where the cost of a miss is high.
What do you do with flaky eval cases?
"I find them by repeated runs (cases that pass 20 to 80 percent of the time), quarantine them so they do not break the build, and fix the cause: an ambiguous rubric, an unfrozen tool, or a genuinely unstable behaviour that deserves its own investigation. A permanently quarantined case is a hole." Follow-up: "How many trials do you need to call a case flaky?" Enough that the interval for its pass rate (module 1) excludes both 0 and 1; five is a start.
How do you gate on cost and latency?
"As separate rules with their own limits, compared with the baseline on the same cases: a cost-ratio threshold fails the build if the candidate is, say, more than 1.5 times as expensive, regardless of quality. A cheaper agent that is slightly worse can be a trade-off a human approves." Follow-up: "Who can override the gate?" A named owner, with a logged reason, never an automatic retry.
Common mistakes
A fixed score threshold on a noisy suite.
Rerunning until green.
Comparing against a baseline from a different model or date without recording it.
One trial per case for a stochastic agent.
No critical cases, so an improved average hides a new catastrophic failure.
A giant slow suite on every commit that engineers skip.
Updating the baseline automatically, so slow decay is never caught.
Letting flaky cases accumulate.
Tuning prompts against the gate's cases until they no longer represent production.
What the example does not do
No real agent or harness; results are simulated pass rates.
A uniform drop in every case; real regressions are concentrated.
Independent cases; shared causes (one broken tool) make wins and losses correlated.
One test: no multiple-comparison correction across slices, no sequential testing.
The CI file is a sketch and was not run.
Optional real experiment (I did not run this): run the same agent twice through your suite with three trials each, and use gate.py to confirm that the gate passes; then degrade the prompt on purpose and confirm that it fails.
Check yourself
A CI gate fails the build when the score drops more than 2 points. The agent did not change. How often does it still fail? (45 percent of the time with 30 cases and one run each; 9 percent with 300 cases and three runs. A paired comparison with a sign test cut false failures to 2 to 5 percent.)
For the curious The price of a safer gate is power
The paired gate caught a true 5-point drop only 7 percent of the time with 30 cases and one run, 45 percent with 100 cases and three runs, and 88 percent with 300 cases and three runs. Fewer false alarms means more missed regressions unless you add cases and repeats. The simulation assumes independent cases; real regressions often hide in a few slices.
What you will use daily at work
Practice
What it looks like
Critical cases that must always pass
One safety or money failure fails the build.
A paired test on the average
Compare candidate and baseline case by case.
Cost and latency limits
A better score that doubles cost can still fail.
Quarantine flaky cases
They inform but do not break the build; then fix or remove them.
Draft: confirm how your CI system stores artifacts and which secrets the eval job may read. The example workflow file was not run.
What you must be able to say (one page)
A regression test suite tracks quality over time and stops a change that makes the agent worse. Store evaluation results with metadata (model version, configuration, timestamp), visualise trends and alert when performance degrades below a threshold. Every time a tool changes, the full suite must be rerun, and mocks or simulators can reproduce rare but catastrophic edge cases.
A fixed threshold on the score is a bad gate. In my simulation, a gate that failed the build when the score fell more than 2 points failed 45 percent of the time on an identical agent with 30 cases and one run each, 23 percent with 100 cases and three runs, and 9 percent with 300 cases and three runs.
A paired gate fixes that. Compare candidate and baseline per case, count wins and losses, and fail only when there is a drop of at least 2 points and statistical evidence (an exact paired sign test at the 0.1 level). Its false-failure rate was 2 to 5 percent across the same sizes.
The price is power. A true 5-point drop failed the build 7 percent of the time with 30 cases and one run, 45 percent with 100 cases and three runs, 88 percent with 300 cases and three runs (naive gate: 65, 82 and 94 percent, but with the false failures above). More cases and repeated trials buy detection; a gate cannot create evidence the suite does not have.
Gate on more than the average:critical cases that must always pass (a safety or money case fails the build even if the average improves), a cost or latency limit, and per-slice checks for important groups.
Quarantine flaky cases (those that pass sometimes but not reliably) so they inform but do not break the build, and fix or remove them.
The example includes a working script:gate.py baseline.json candidate.json --critical a,b exits 1 when the gate fails (tested), plus an example GitHub Actions file that I did not run.
Honest limits: the simulation assumes independent cases and Beta-distributed pass rates; real regressions are often concentrated in a few slices.