Part 1 of 4 · 1 min read
Overview
Agents act, so an incident is not only downtime but wrong actions that already happened: money moved, emails sent, data shown to the wrong tenant. Detection speed and the ability to stop the agent quickly matter more than for a read-only API. Interviewers ask "your agent started issuing wrong refunds, what do you do?"; the answer is the lifecycle above, with a kill switch first.
About the example, and what is real
Real: the alert rules (threshold, multi-window burn rate, cost-rate), the scoped kill switches with an audit log, the severity function, the incident clock metrics, the runbook linter, the post-mortem checker, and the 21 tests in code/mini-incident. Assumed: the traffic (200 requests per minute), the noise (random 2-minute spikes at 10% errors with probability 0.004 per minute), the incident shapes, the refund-bug rates and times, and the severity thresholds. Not real: no live monitoring, paging system or real incident; the numbers show how rules behave on my simulated data, not what your system will do.