Definitions

Definitions

Replay

Results

1. Restart cost grows with crash rate; journalled cost stays near the no-crash cost.
2. A journal does not make effects exactly-once; idempotency keys do.
3. Replay must be deterministic, so record non-deterministic values.
4. Waits and compensations fit the same journal.

Design rules

• Journal every expensive or non-repeatable step before moving on; store results, not just "done".
• Give each effect an idempotency key derived from run id and step name, and use it downstream.
• Keep orchestration code deterministic; put LLM calls, I/O, time and randomness inside recorded steps or Ctx-style recorded values.
• Version workflows; refuse to resume a journal against a different structure, or run old versions to completion.
• Model waits as journalled suspensions, with timeouts and escalation (Level 10 module 9).
• Plan compensations for effects that matter; make them idempotent; log them.
• Heartbeat and lease (module 2) so a dead worker's run is picked up by another.
• Cap total run time, steps and cost even for durable runs (Level 9 budgets).
• Test by killing workers at random positions in staging (chaos testing).
• Choose build or buy deliberately: a journal table plus a queue covers small cases; durable-workflow engines (Temporal and similar) or framework checkpointers cover the rest.

Interview questions, with the follow-ups

Common mistakes

• Restarting from scratch and repeating spend and effects.
• Journalling "step done" without the result, so replay cannot continue.
• No idempotency keys on effects, trusting the journal alone.
• Non-deterministic orchestration code (time, random, unordered iteration).
• Resuming old journals against new code.
• Holding a worker while waiting for a human.
• Compensations that are not idempotent or never tested.
• Unbounded journal growth on very long runs.

What the example does not do

• In-memory journal, single process: no real storage, ordering, concurrent writers or exactly-once journal writes.
• Crash positions are injected, not real process kills; partial writes and corrupted journals are not simulated.
• Probabilities and costs are assumptions.
• Idempotency is assumed to be honoured downstream; many real APIs do not support keys.
• No history versioning, patching or migration, only a refuse-to-resume hash.
• Compensation is a toy: real compensations can fail or be impossible.
• Optional real experiment (I did not run this): run a real agent under a durable engine or your own journal, kill the worker at random during runs in staging, and compare cost, duplicate effects and completion with a restart-on-failure setup.

For the curious The dangerous gap is between the effect and the journal write

Journalling alone still repeated 35 effects, from crashes after the effect and before the journal write. An idempotency key (run id plus step name) on each external effect brought that to 0. Workflow code must be deterministic on replay, so record clock and random values in the journal, and refuse to resume if the workflow structure changed. In-memory journal, single process, assumed probabilities.

What you must be able to say (one page)

• Durable execution means a run survives the process that was executing it. The worker can be killed, redeployed or evicted mid-task and the run continues from where it was, not from the start. The mechanism is a journal (checkpoint log) of completed steps: on restart, completed steps are replayed from the journal (their recorded results reused, nothing re-executed) and execution continues at the first unfinished step.
• Why it matters, in simulation (400 runs of a 7-step agent with 3 effects, LLM cost 8.0 without crashes, 8 percent crash chance per step, assumed): restarting from scratch cost 12.43 per run; journalled resume cost 8.22. At crash probability 0.2 per step the gap widens to 25.5 versus 8.53. Restarting also repeated side effects 243 times (charge, email, CRM update).
• The dangerous window is "effect done, journal not written." Journalling alone still repeated 35 effects (crash after the effect, before the journal write). Adding an idempotency key (run id plus step name) to each external effect gave 0 duplicates. Journalled LLM steps can still be re-run if the crash hits before their journal write, which is why the cost is 8.22, not 8.0.
• Workflow code must be deterministic on replay. If orchestration code reads the clock or a random number live, replay can take a different path than the original run and disagree with the journal; my engine raises a non-determinism error when the journal's step name differs from the step the code would run. The fix is to record such values in the journal on first execution and replay them (my Ctx.random()), and to keep non-determinism (LLM calls, I/O) inside recorded steps.
• Changing a workflow while runs are in flight is a hazard. A journal written for steps A, B, C does not fit a new A, B2, C. My engine compares a structure hash and refuses to resume; production systems version workflows and keep old versions until old runs finish (module 9).
• Waits are states, not sleeping workers. A human approval or timer step suspends the run with no worker held; when the signal arrives the run resumes from the journal (ties to Level 8 human-in-the-loop and Level 10 approvals).
• Sagas for failures you cannot retry away. If a later step fails permanently after earlier effects, run compensations in reverse order (refund the card, send a retraction). Compensation is a best effort and itself must be idempotent.
• Granularity is a trade-off: journal per LLM call or tool call (cheap to lose little, more writes), versus per phase (fewer writes, more rework). Journal what is expensive or non-repeatable.
• Honest limits: in-memory journal, single process, crash points injected at three positions per step, assumed probabilities and costs; no real engine, storage, concurrency or versioning of history.