Part 4 of 6 · 3 min read
Other failures and designing for failure
Stale knowledge, drift, over-confidence and tool errors, then five rules that keep a failure from spreading.
- Stale knowledge. Training stops at a cutoff date. Anything later (a new library version, today's date, current prices) is unknown or wrong. Give the model the current information or a tool to fetch it, and put today's date in the prompt.
- Instruction drift. In a long conversation, early instructions lose weight against recent text. Keep rules in the system prompt, restate the critical ones, and trim history (module 6).
- Over-confidence. A fluent tone does not signal reliability. Do not use "the model sounded sure" as evidence.
- Tool and format errors. Malformed JSON, wrong argument names, calls to tools that do not exist. Constrain the output (module 4) and validate with Pydantic (Level 0, module 1).
Designing for failure
- Validate every output before acting on it: schema, ranges, allowed values.
- Verify with a different mechanism where you can: code checks numbers, search confirms facts, tests confirm code.
- Keep a person in the loop for actions that cannot be undone (sending, paying, deleting).
- Fail loudly and stop early. When a check fails, retry once with the error message, then escalate. Do not let a bad value flow on.
- Log everything (prompt version, inputs, outputs, checks) so a failure can be replayed and added to your evaluation set (module 5). Every real failure you find is a free test case.
How errors compound
If every step must be right, end-to-end success is the per-step accuracy raised to the number of steps. At 95% per step, 5 steps give 0.774, 10 steps give 0.599 and 20 steps give 0.358. Even a very good model needs checks between steps. (Easy 1 asks you to build the full table for three accuracy levels.)
For the curious Why a check between steps beats a check at the end
A single check at the end catches an error only if it is still visible then. Errors early in a chain are often buried under later steps, which treat the wrong value as true. A check right after the step that produced it sees the error while it is small and easy to explain (Hard 3 simulates this).
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Grounded answers with citations | The model must quote the source, and code checks that the quote is in the source. |
| Output validation | Every model output is parsed against a schema before any action is taken. |
| Probes in the evaluation set | Needle, pushback and wording probes run on the real model on a schedule. |
| Human approval for risky actions | Anything irreversible waits for a person (Level 10). |
| Failure logging | Real failures are saved with their inputs and become new test cases. |
| Self-consistency for key answers | Several samples are compared and a disagreement triggers a retry or an escalation. |
Draft: to be checked against current practice and provider guidance on grounding and citations before this page goes live.
Interview angle
Answer in your head first, then open it.
Why do LLMs hallucinate?
They are trained to produce plausible continuations, not verified truths, and they fill gaps with likely-sounding text.
How do you reduce hallucination in an application?
Ground answers in retrieved sources, require evidence, validate with code where possible, and allow "I do not know".
What is lost in the middle and how would you test for it?
Weaker use of information in the middle of long inputs; a needle-in-a-haystack probe at several positions.
What is sycophancy and where does it come from?
Agreeing with the user over the truth, encouraged by preference tuning. Mitigate with neutral prompts and independent critics.
A 10-step agent has 95% accuracy per step. What is the end-to-end success rate?
About 60%, which is why checks between steps matter.