Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-rollout.
Done when: you report the sample size for a 3-point change on a 70 percent baseline.
Easy 2: Sample size by hand (30 minutes)
Compute the per-arm size for a 90 percent baseline and a 2-point change.
Done when: you match n_per_arm (3,213).
Hint
use the normal approximation with the pooled variance, z of 1.96 and 0.84.
Easy 3: Pick guardrails (30 minutes)
List five guardrail metrics for a customer-support agent with a refund tool.
Done when: each has a rollback threshold.
Medium
Medium 1: Canary stage design (3 hours)
Change the stage sizes.
Done when: you find the smallest stages that catch a 2-to-3 percent regression at stage 2 at least 50 percent of the time.
Hint
the stage population is the stage fraction times total traffic.
Medium 2: Sequential test (4 hours)
Implement an alpha-spending rule and compare with Bonferroni.
Done when: you report false positives and average sample size for both.
Hint
an O'Brien-Fleming-style rule spends little alpha early.
Medium 3: User-level correlation (3 hours)
Give each user 10 correlated requests.
Done when: you show that per-request randomisation understates the variance.
Hint
the effective sample size is n divided by the design effect.
Hard
Hard 1: Real shadow replay (8 hours)
Replay logged inputs through two versions of a real agent with tools mocked. (I did not run this, so your numbers are the ground truth.)
Done when: you report paired differences with intervals.
Hint
disable or mock every tool that writes.
Hard 2: Automatic rollback (6 hours)
Build a canary controller with a version flag and a rollback rule. (I did not run this.)
Done when: a test shows rollback within a stated number of requests of a simulated regression.
Hard 3: A/B on an agent (8 hours)
Run an A/B test on a real agent change with a pre-registered sample size. (I did not run this.)
Done when: you report the effect, interval and guardrail results.