Part 1 of 4 · 1 min read
Overview
Agent jobs are long, expensive, rate-limited by model providers and sometimes poisonous (a prompt that always crashes the parser). A queue with leases, backoff, a DLQ and fairness is the difference between a spike and an outage. Interviewers ask "how do you process agent tasks reliably at scale?"
About the example, and what is real
Real: the queue (leases, nack with delay, attempt counting, dead-lettering, round-robin tenant scheduling), backoff with jitter, and the 19 test cases in code/mini-queue. Assumed: 5 percent crash rate with a 50/50 split between before and after the side effect, 50 requests per second capacity, one-second jobs, tenant sizes, retry-forever policy for the no-DLQ case. Not real: no real broker (SQS, RabbitMQ, Redis, Kafka, a database queue), no network, no clocks; one process.