Which errors to retry, how to wait between attempts (backoff, jitter, the server's own hint) and how to respect a deadline.
Decide what to retry
Transient error
A failure that may disappear if you try again later: a timeout, a rate limit (HTTP 429), a server error (5xx), a dropped connection. Errors caused by your request fail the same way every time: bad JSON (400), no permission (401, 403), not found (404), invalid fields (422).
Status
Meaning
Retry?
timeout, 408
Took too long
Yes
429
Too many requests
Yes, after the wait the server asks for
500, 502, 503, 504
Server trouble or overload
Yes
409
Conflict with the current state
Only if you know it is temporary (a lock or an operation in progress). The example retries it, as some SDKs do by default.
400, 422
Your request is invalid
No: fix the request
401, 403
Not allowed
No
404
No such thing
No
The function is_retryable encodes this table, and a test runs eleven status codes through it. A bug in your own code (ValueError) is never retried.
Two cautions. First, a timeout does not tell you the request failed. The server may have done the work, and only the reply was lost (section 3). Second, many provider SDKs already retry for you. Find out what yours does by default, so you do not stack three layers of retries on top of each other and turn three attempts into twenty-seven.
Check yourself
Your code raises KeyError inside the function you are retrying. Should the retry loop try again? (No. It is a bug in your code, and repeating it wastes time. Only errors that may disappear are retried.)
Retry with backoff and jitter
Three retry policies against the same four failures. Real output of the example code.
Exponential backoff
Wait longer after each failure, for example 1 s, 2 s, 4 s, 8 s, up to a cap. It gives an overloaded service room to recover.
Jitter
Randomise the wait, for example choose a random time between 0 and the computed delay. If many clients fail together, plain backoff makes them all retry at the same instants and fail together again. Jitter spreads them out. Level 1, module 6 showed the formulas.
retry_call(fn, policy, clock) implements the rules, each with a test:
Only retry retryable errors, and give up at once on a 400 (one call made).
Backoff doubles and is capped. With a base of 1 and a cap of 5 the waits are 1, 2, 4, 5, 5, 5.
Obey the server. If a 429 carries retry_after = 7, wait at least 7 seconds, not the shorter computed delay. Real servers send this in a Retry-After header as a number of seconds or as a date, so parse both.
Cap the attempts and return the list of errors, so you can log them.
Respect a deadline. If the next wait would push the total past the time you can afford, stop now. A user waiting for an answer does not benefit from a perfect retry that arrives after they gave up.
Refuse max_attempts below 1. The first version accepted 0 and returned an empty "success" with value None.
The function returns a report (value, attempts, total wait, errors, why it gave up) rather than only a value. Reports are what you log and graph. In the demo, a 429 with retry_after=2 and then a timeout gives waits of 2.0 and 1.0 seconds (the first obeys the server, the second is the computed 0.5 x 2) and success on attempt 3.
Retries multiply load and cost
Three attempts mean up to three times the traffic during an outage, which is exactly when the service is weakest. Keep attempts small (three or four), add jitter, and use a circuit breaker (section 4).
Timeouts
Always set a timeout on every network call. Without one, a stuck connection waits forever and your agent hangs. Use two: a short one to connect, and a longer one for the response. A long generation can take a minute. With streaming, apply a timeout to the gap between events (module 2). The total of all attempts must fit inside one overall deadline for the user's request. Note that the example's deadline only counts the waits it chooses, not how long the call itself takes. A real client must also stop a call that runs past the deadline (exercise Medium 2).
Check yourself
With base 0.5 s, cap 8 s and no jitter, what are the waits before attempts 2 to 7? (0.5, 1, 2, 4, 8, 8. Total 23.5 s.)