Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-queue.
Done when: you report lost and duplicate counts for the three delivery modes.
Easy 2: Lease length (30 minutes)
Make the lease shorter than the job time in a test.
Done when: you show duplicate execution while the first worker is still running.
Hint
lease 5 s, job 8 s; the second receive happens at t = 5.
Easy 3: Classify errors (30 minutes)
List five errors from an agent run and mark each retry or do-not-retry.
Done when: each has a reason.
Medium
Medium 1: Heartbeats (3 hours)
Add lease extension while a worker is alive.
Done when: a long job is not duplicated but a crashed worker's job is redelivered.
Hint
heartbeat by setting lease_until = now + lease for the leased message.
Medium 2: Weighted fairness (3 hours)
Give tenants weights and a per-tenant concurrency cap.
Done when: a weight-3 tenant gets about three times the throughput under load.
Hint
pick the tenant with the lowest served / weight.
Medium 3: Redrive (3 hours)
Add a function that moves DLQ items back to ready after a fix.
Done when: attempts reset and a fixed handler completes them.
Hard
Hard 1: Real broker crash test (8 hours)
Run workers against SQS, Redis or Postgres and kill them mid-job. (I did not run this, so your numbers are the ground truth.)
Done when: you report lost and duplicate effects for ack-first and ack-last.
Hint
Postgres SELECT ... FOR UPDATE SKIP LOCKED is the smallest real queue.
Hard 2: Retry storm (6 hours)
Reproduce a retry storm against a rate-limited service with and without jitter. (I did not run this.)
Done when: you plot requests per second and time to recover for each.
Hard 3: Autoscaling (6 hours)
Scale workers on age of oldest message versus depth. (I did not run this.)
Done when: you compare queue age during a burst and cost for each rule.