Crash after the work of a step, before it is recorded; 2,000 runs per cell; six model calls without crashes (five steps plus one final call), up to six attempts:
Crash chance per step
Loop: calls / duplicates / finished
Graph + checkpoint
Graph + checkpoint + idempotency key
0%
6.0 / 0.00 / 100%
6.0 / 0.00 / 100%
6.0 / 0.00 / 100%
5%
6.9 / 0.17 / 99.95%
6.3 / 0.11 / 100%
6.3 / 0.00 / 100%
15%
9.1 / 0.50 / 96.9%
6.9 / 0.34 / 99.7%
6.9 / 0.00 / 99.7%
30%
11.6 / 0.56 / 67.5%
8.0 / 0.81 / 94.5%
8.0 / 0.00 / 94.5%
Model calls per run and duplicate side effects per run, for three structures.
Checkpointing saves model calls and lets more runs finish. At 30 percent crashes the loop used 45 percent more calls and finished 27 points fewer runs.
Checkpointing does not make a step exactly-once. A crash after the work and before the checkpoint repeats that step on resume.
Idempotency removes the duplicates but saves no calls. A loop with idempotency keys had 0.00 duplicates and the same 11.6 calls.
Read the 30 percent duplicates carefully. The graph shows more duplicates per run (0.81) than the loop (0.56). This is because it finishes far more runs (94.5 against 67.5 percent) and so executes more steps, not because it is worse. Always read duplicates next to completion.
Coarser checkpoints cost calls. The tests show that checkpointing every three steps needs more calls than every step; the price of a checkpoint is storage and latency, which this model does not count.
Handoff against agent-as-tool (assumed sizes; total input tokens for triage plus k specialists):
Specialists k
1
2
3
5
8
Handoff
4,500
7,500
11,000
19,500
36,000
Agent-as-tool
4,750
7,650
10,700
17,250
28,200
History passing grows with the square of the chain, while the brief-and-summary pattern grows linearly. The crossover is at k = 3 with these numbers; with larger briefs or summaries it moves.
Choosing
Start without a framework if the loop is under a few hundred lines and you need to see every call; building first lets you debug what a framework would hide. The OpenAI guide also says to start with one agent.
Choose by the hardest requirement. Durable multi-day runs with approvals: a graph with a durable store. A coding or file-working agent with tools: an SDK built around that loop. Several specialists with distinct policies: handoffs or agent-as-tool, with traces to read.
Re-evaluate on a schedule. Frameworks change fast, and I found the Anthropic multi-agent post modified in October 2026; treat any comparison table as dated.
Isolate the framework behind your own interface for model calls, tools and state, so that switching is a rewrite of an adapter. A good exercise is the same agent in two frameworks.