Part 3 of 6 · 2 min read
Idempotency keys
A lost reply makes a retry repeat the action. A unique key sent with every attempt prevents it.
Idempotent
An operation that has the same effect whether it is done once or many times. Reading is idempotent. "Set the status to shipped" is idempotent. "Add 10 dollars to the balance" and "send an email" are not.
The dangerous case is a lost reply. Your request reached the server, the action happened, but the answer never reached you. A retry would do it again. The fix is an idempotency key: a unique ID you generate for the intended action and send with every attempt. The receiver remembers the result for that key and returns it instead of acting again.
IdempotentExecutor.run(key, action) shows the logic. Two tests tell the story:
- With a key: the charge happens, the reply is lost, the retry returns the saved result, and the total number of charges is 1.
- Without a key: the same sequence charges twice.
Rules:
- Generate the key once per intended action, before the first attempt, and reuse it for every retry. A new key per attempt defeats the purpose.
- Many payment and messaging APIs accept an idempotency key header. Use it. For your own tools, store keys in a database together with the result.
- Make tools idempotent by design where you can (module 4): "ensure a ticket exists with this ID" rather than "create a ticket".
For the curious What the small example leaves out
The example keeps keys in a dictionary, so a restart forgets them (Medium 3 moves them to a database). It also does not handle two requests with the same key at the same time: both would run the action, because neither has finished yet. Real services lock the key while the first request is in progress. And a real service should reject a repeated key that arrives with a different request body, since that is a bug in the caller.
Check yourself
A run makes 15 calls and each fails with probability 2%. What is the chance that at least one fails? What if each call gets three attempts in total? (About 26%, because 1 - 0.98^15. With three attempts a call fails only when all three do, 0.02^3, so about 0.01%. This assumes failures are independent, which is false during a real outage. That is why the circuit breaker exists.)