Stop calling a service that is down, switch to a backup, and how the pieces fit in an agent run.
Circuit breaker
Three states. While open, calls fail immediately without touching the service.
If a service is down, retrying every call from every user makes things worse and makes every request slow. A circuit breaker watches failures and, after a threshold, stops calling for a cooldown period and fails fast. After the cooldown it lets a trial call through. Success closes the circuit, and failure reopens it.
The demo shows the facts. After three failures the service has received exactly 3 calls. The next three attempts reach the service zero times and fail at once with circuit open. After the cooldown the state is half_open, and the trial call decides what happens next.
For the curious A bug the first version had
The first breaker counted every exception as a failure. Three requests with a bad payload (a 400, which is the caller's fault) opened the circuit, and then every valid request from every user failed fast for 30 seconds. A breaker must only count errors that say the service is unhealthy, which is the same list as is_retryable. A test now sends ten bad requests and checks that the circuit stays closed. The example is also simplified for one thread: after the cooldown it does not limit how many trial calls run at once, which a production breaker does.
Failing fast is a feature. The agent can switch to a fallback at once instead of waiting through timeouts.
Fallbacks
with_fallbacks([("primary", call_a), ("backup", call_b)]) tries each in order and returns which one worked. What can sit behind a fallback:
Another model or provider (module 7 builds the layer that makes this easy).
A smaller, cheaper model, accepting lower quality.
A cached or default answer for non-critical features.
A graceful message ("I could not complete that, try again in a minute") with the partial progress saved.
Record which fallback served each request. If the backup quietly serves 40% of traffic, you have a problem you cannot see. Also test quality: a fallback model may need different prompts (Level 3).
Putting it together for an agent
for each step of the run:
result = breaker.call( lambda: retry_call( lambda: model_call(...), policy ) )
if it fails: use a fallback, or stop with a clear error and saved state
for each tool with side effects:
run it with an idempotency key
Also persist the conversation after each step, so a crash can resume instead of starting over, and put a hard cap on steps, tokens and time per run (Level 1, module 6).
What you will use daily at work
Practice
What it looks like
SDK retry settings
You set the maximum retries and timeouts on the provider's client instead of writing a loop.
A retry library
Libraries such as tenacity wrap functions with backoff, jitter and stop rules.
Idempotency keys on side effects
Payments, emails and tickets carry a key derived from the run and step.
Dashboards and alerts
Retry counts, breaker state and fallback share are graphed, and a sudden rise pages someone.
Draft: confirm default retry behaviour, timeouts and rate-limit headers against your provider's current documentation before relying on them.
Interview angle
Answer in your head first, then open it.
Which errors do you retry?
Timeouts, 429 and 5xx. Never client errors such as 400, 401 or 404.
Why jitter?
So many clients that fail together do not retry together.
What is an idempotency key and when do you need one?
A unique ID reused across retries, so the receiver does the action once. You need it for any retried action with side effects.
What does a circuit breaker do, and what should it count?
After repeated failures it stops calling for a cooldown, failing fast, then tests with a call. It counts only failures that mean the service is unhealthy, not bad requests.
A request timed out. Did it happen?
You cannot know. Design for both outcomes with an idempotency key or a status check.
Why can nested retries be dangerous?
Layers multiply: 3 attempts inside 3 attempts is 9 calls.