Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-incident.
Done when: you report detection delay and false pages for both rules.
Hint
the threshold slow-burn catch is chance; try other seeds.
Easy 2: Severity (30 minutes)
Classify five incidents from your own system with severity or your own rules.
Done when: each has a severity and a one-line reason.
Easy 3: Lint a runbook (30 minutes)
Write a runbook for "agent cost per minute above budget" and run lint_runbook.
Done when: it passes with a concrete mitigation.
Medium
Medium 1: Tune the alerts (3 hours)
Change the burn-rate windows and factors and re-run evaluate.
Done when: you show a table of delay versus false pages and choose a setting.
Hint
a longer long window gives fewer false pages and more delay.
Medium 2: Kill switch in your agent (3 hours)
Add a tool and tenant switch to an agent you wrote, checked before every tool call.
Done when: a test shows the switch blocks a call, fails closed and logs who and why.
Hint
check the switch inside the tool dispatcher, not in the prompt.
Medium 3: Post-mortem (3 hours)
Write a blameless post-mortem for the simulated refund incident.
Done when: check_postmortem returns no problems and the actions have owners and dates.
Hard
Hard 1: Game day (8 hours)
Run a game day on a staging agent with a colleague. (I did not run this, so your numbers are the ground truth.)
Done when: you report time to detect, mitigate and resolve and fix two things it exposed.
Hard 2: Real traffic burn rate (6 hours)
Replay a week of your real error series through both rules. (I did not run this.)
Done when: you report pages for each and which were actionable.
Hard 3: Wrong-action recovery (8 hours)
Build a tool that lists every action by a release id and a time window from your audit log and proposes compensations. (I did not run this.)
Done when: you can answer "what did release X do between 10:00 and 10:30?" in one query.
Hint
tag every tool call with the release id and run id.