Covers: Definitions; The lifecycle; Results; Design rules; Interview questions, with the follow-ups; Common mistakes; What the example does not do.
Definitions
Queue, worker, broker
Queue: a buffer of jobs waiting to be processed.
Worker: a process that takes jobs and runs them.
Broker: the service that stores and hands out messages.
Ack, nack, lease
Ack: the worker confirms the job finished.
Nack: the worker reports failure, optionally with a delay before retry.
Lease (visibility timeout): how long a received job is hidden from others before being offered again.
At-most-once, at-least-once, idempotent
At-most-once: never duplicated, may be lost.
At-least-once: never lost, may be duplicated.
Idempotent: running twice has the same effect as once.
Backoff, jitter
Exponential backoff: wait longer after each failure.
Jitter: add randomness so clients do not retry together.
Dead-letter queue, poison message
Dead-letter queue (DLQ): where jobs go after too many failed attempts.
Poison message: a job that fails every time.
Backpressure
Backpressure: slowing or rejecting new work when the system is full.
The lifecycle
Lease, ack, redeliver, dead-letter.
Results
Scenario (simulated)
Outcome
Ack first, 5% crashes
99 lost, 0 duplicate effects
Ack last
0 lost, 65 duplicate effects
Ack last + idempotent effect
0 lost, 0 duplicates
1000 retries, no jitter
peak 1000/s, 9,500 rejected, 962 s to clear
1000 retries, full jitter
peak 720/s, 2,283 rejected, 69 s to clear
Poison jobs, retry forever
0 of 40 good jobs done in 2000 s
Poison jobs, DLQ after 3
40 of 40 done by 26 s; 4 parked
Tenant B p95, FIFO vs round-robin
101 s vs 4 s
Delivery, retry storms, poison jobs and fairness.
Pick at-least-once and make the work idempotent; at-most-once silently loses jobs.
Jitter shortens recovery but does not by itself prevent the first spike.
A DLQ is a safety valve, not a bin: someone must read it.
Fairness is a scheduling decision, not a worker-count decision.
Design rules
Ack after the work, with an idempotent handler keyed by job id.
Set the lease above p99 run time and heartbeat to extend it for long runs.
Retry transient errors with backoff and jitter; do not retry permanent ones (bad input, failed validation).
Cap attempts and dead-letter, keeping payload, error and attempt history; alert on DLQ depth and age.
Schedule per tenant with round-robin or weights, and per-tenant concurrency caps.
Bound queue depth and apply backpressure; scale workers on queue age.
Respect downstream rate limits with a shared limiter, not per-worker guesses.
Separate queues by job class (fast chat versus slow research) so slow jobs do not block fast ones.
Make jobs small and resumable (module 3) so a lease expiry loses little work.
Expose queue metrics: depth, age of oldest, attempts, DLQ count, time in queue versus run time.
Interview questions, with the follow-ups
At-most-once versus at-least-once: which do you choose?
"At-least-once with idempotent handlers. In my simulation acking first lost 99 of 2000 jobs; acking last lost none but ran 65 side effects twice; an idempotency check removed the duplicates. Exactly-once is built from at-least-once plus dedupe." Follow-up: "How do you make an email send idempotent?" Record the job id before sending (or pass an idempotency key to the provider) and skip if present; accept that a crash between send and record can still duplicate, so prefer provider-side keys.
What is a dead-letter queue for?
"To stop poison messages from consuming workers forever. After a maximum number of attempts the job is parked with its error for investigation. In my test, retrying forever left 40 good jobs undone while dead-lettering finished them by 26 seconds." Follow-up: "What do you do with DLQ items?" Alert, inspect, fix the cause or data, then redrive; add the case to the regression suite.
Why add jitter to retries?
"Without it clients that failed together retry together. With 1000 clients and 50 requests per second capacity, no jitter rejected 9,500 requests and took 962 seconds to clear; full jitter rejected 2,283 and cleared in 69 seconds. It does not fix the first spike, so I also use a wider initial window or admission control." Follow-up: "What else helps?" Circuit breakers and shared rate limiters (module 6).
One customer floods the queue. What happens to the others?
"With FIFO they wait behind the backlog: in my simulation the small tenant's p95 was 101 seconds. Round-robin per tenant brought it to 4 seconds without hurting the big tenant. I add per-tenant concurrency limits and weights." Follow-up: "How do you stop one tenant using all workers?" Concurrency caps per tenant and per job class.
Common mistakes
Ack before work, losing jobs on crashes.
Non-idempotent handlers behind an at-least-once queue.
Lease shorter than the run time with no heartbeat.
Retrying permanent errors and rate-limit responses immediately.
No jitter, so every retry lands together.
No DLQ, or a DLQ nobody watches.
One shared FIFO for all tenants and job types.
Scaling workers on queue depth only, ignoring age.
Unbounded queues, turning an outage into a cost event.
What the example does not do
One process, tick-based, no real broker, network, persistence or clock drift.
The retry-storm model is simplified: all clients start failed together and clients that are rejected retry once more per attempt.
Poison results depend on FIFO order and retry delay; a longer delay or per-attempt demotion would let good jobs through earlier.
No exactly-once claims, no partitioned ordering, no transactional outbox.
Optional real experiment (I did not run this): run a real queue (SQS, Redis or Postgres SKIP LOCKED), kill workers at random while jobs run, and count lost jobs and duplicate effects for ack-first and ack-last.
Check yourself
A worker crashes after sending an email but before acknowledging the job. What happens next, and what stops a second email? (The lease expires and the job is delivered again. An idempotent effect, keyed by job and step, stops the duplicate; in the simulation at-least-once delivery with an idempotent effect gave 0 lost and 0 duplicates.)
For the curious Retrying forever lets one bad job starve the rest
With 4 poison jobs and 40 good ones on 2 workers, retrying forever left all 40 good jobs undone after 2000 s. Dead-lettering after 3 attempts finished them by 26 s. Likewise, under first-in first-out a small tenant waited 101 s at p95, against 4 s with round-robin per tenant. These are tick-based simulations with assumed crash rates and job times, not a real broker.
What you will use daily at work
Practice
What it looks like
Ack after the work, make the effect idempotent
At most once loses jobs; at least once duplicates them.
Add jitter to retry backoff
Full jitter cut the retry clearing time from 962 s to 69 s.
Dead-letter after a few attempts
And alert on the dead-letter queue.
Schedule fairly per tenant
Cap concurrency per tenant and per downstream.
Draft: confirm which queue your platform uses and how it implements visibility timeouts, ordering and dead-letter queues.
What you must be able to say (one page)
A queue decouples accepting work from doing it. The API enqueues a job (module 1) and returns; workers pull jobs, run the agent and report. This absorbs bursts, lets you scale workers independently and gives you a place to apply retries, limits and priorities.
Delivery semantics decide what breaks. Simulation (2000 jobs, 5 percent of worker attempts crash, one side effect each, assumed): ack before work (at-most-once) lost 99 jobs; ack after work (at-least-once) lost 0 but ran 65 duplicate side effects (crash after the effect, before the ack); at-least-once plus an idempotent effect gave 0 lost and 0 duplicates. Rule: use at-least-once and make handlers idempotent (dedupe by job id, or by an idempotency key on the downstream call).
Leases (visibility timeouts) detect dead workers. A received message is invisible to others for a lease period; if not acked in time it becomes ready again and its attempt count rises. A lease shorter than the job's run time causes duplicate runs while the first worker is still alive; renew the lease (heartbeat) for long agent runs.
Retry with exponential backoff and jitter. 1000 clients retry after an outage against a service that accepts 50 requests per second: with no jitter the peak was 1000 requests in one second, 9,500 requests were rejected and clearing took 962 s; with full jitter the peak was still 720 (the first window is only 2 s wide), but rejections fell to 2,283 and clearing to 69 s. Jitter spreads later retries but does not remove the first spike; widen the initial window or add admission control.
Dead-letter queue (DLQ) for poison messages. A job that always fails will otherwise be retried forever. With 4 poison and 40 good jobs on 2 workers, retrying forever left all 40 good jobs undone after 2000 s (3,998 wasted executions: the retried poison jobs are first in FIFO order and take both workers), while dead-lettering after 3 attempts finished all 40 good jobs by 26 s (20 s with no poison jobs at all), parking the 4 bad ones for a human. Keep the message, last error and attempt history in the DLQ and alert on its depth.
Fair scheduling protects tenants. Tenant A queues 500 one-second jobs; tenant B queues 10 one second later; 5 workers. Under FIFO, B's p95 latency was 101 s; under round-robin per tenant it was 4 s, and A's barely changed (96 s versus 98 s). Per-tenant concurrency caps and weights are the production version (module 7).
Backpressure and limits: bound the queue, reject or delay when full, cap concurrency per tenant and per downstream (LLM rate limits, module 6), and scale workers on queue age, not just depth.
Honest limits: in-memory, tick-based, one process; assumed crash rates and job times; no real broker, ordering guarantees across partitions, or visibility-timeout drift.