Definitions

Definitions

Two detectors

Results

Faithfulness check on 22 claims against the context. A claim is flagged when it is not supported. (13 false claims, 9 true claims.)

By kind of claim (share judged correctly):

(The last row is correct only because the paraphrase has low overlap.) The overlap threshold made no difference between 0.4 and 0.9 on this set, which shows the set is too small to tune on.

Self-consistency (4,000 questions, 70 percent known; flag when agreement is below 100 percent): unknown facts flagged at k=2: 83 percent; k=3: 97; k=5: 99.7; k=10: 100; known facts wrongly flagged: 0 in every case. At k=5 with a share of unknown facts answered with one consistent wrong answer: 0 percent: 100 percent flagged; 30: 68; 60: 39.

1. A lexical check finds invention and gross contradiction cheaply; it fails on paraphrase and on small value changes unless you add rules, and every rule costs false alarms.
2. Numbers and negations are the highest-value special cases for business text (amounts, dates, policies); check them with code against the source, not with a model.
3. Consistency needs several samples, so cost scales with k, and it helps only for errors that vary.
4. Prevention beats detection: retrieval with citations, abstention and tool-verified facts reduce the problem you have to detect.

Design rules

• Give every factual claim a source (a passage, a tool result) and show it to the user or the checker.
• Extract and verify numbers, dates, names and identifiers by code against the source.
• Use an entailment model or a calibrated LLM judge for paraphrase, and measure its error rates (module 4).
• Allow and reward abstention: "not in the provided documents" is a correct answer.
• Use self-consistency selectively (high-stakes answers, no context available), with k of 3 to 5, and treat low agreement as a trigger for abstention or review.
• Measure on held-out claims from real traces (module 2), sliced by kind of claim.
• Log unsupported claims in production and sample them for review (module 8).

Interview questions, with the follow-ups

Common mistakes

• Tuning detector rules on the test claims.
• Overlap-only checks that pass changed numbers.
• Treating a high consistency score as correctness.
• No abstention path, so the model must answer.
• Checking against retrieved text without checking the retrieval.
• Claims not extracted: scoring a whole paragraph as one unit.
• Ignoring the false-alarm cost of the checker.
• Fine-tuning to inject facts that change.

What the example does not do

• No real answers, and a hand-written, templated claim set of 22.
• Rules tuned on the test set; no held-out evaluation.
• No entailment model or LLM judge, so paraphrase is a false alarm by construction.
• A two-state model of knowledge in the consistency experiment; real uncertainty is graded.
• Independent samples; real samples at low temperature are correlated.
• Optional real experiment (I did not run this): collect 100 answers with their retrieved contexts, label each claim by hand, and compare overlap, number and negation checks, an entailment model and a judge on held-out claims.

For the curious Self-consistency finds guesses, not confident errors

In the simulation, unknown facts were flagged 83 percent of the time with 2 samples, 97 with 3 and 100 with 5, and known facts raised no false alarms. A model that is confidently wrong gives the same answer every time, so consistency cannot flag it. The rules in this module were written while looking at the 22 test claims, so their 100 percent figures are optimistic.

What you must be able to say (one page)

• Define it before you detect it. A narrow definition: a model has hallucinated if its generated text is not grounded in its training data or its input context. For agents, the useful split is unfaithful to the context (contradicts or goes beyond the retrieved text or tool output) and unsupported by the world (no context to check; the model invented it). A knowledge quadrant helps: known knowns, unknown knowns, known unknowns and unknown unknowns, where the dangerous cell is not knowing that it does not know.
• Two detectors: a faithfulness check (split the answer into claims and test each against the trusted context) and a self-consistency check (sample the same question several times; facts the model knows repeat, guesses vary).
• A cheap lexical check has predictable blind spots. On my 22 hand-written claims about a policy text (13 false, 9 true): word overlap alone flagged 54 percent of false claims; it passes changed numbers (same words, different figure) and flipped meaning (only 33 percent of negation flips caught). Adding a number check raised recall to 85 percent; adding a negation check to 100 percent, but at a 22 percent false-alarm rate because true paraphrases ("two weeks after we receive the item") share too few words.
• Those 100 percent figures are optimistic: I wrote the rules while looking at these 22 claims. A real check needs a held-out set, and an entailment model or a calibrated LLM judge (module 4) for paraphrase.
• Self-consistency flags guesses, not confident errors. In my simulation of a model that either knows a fact (always the same answer) or guesses among six wrong answers, the share of unknown facts flagged was 83 percent at 2 samples, 97 at 3, 100 at 5, with no false alarms on known facts. But if the model gives the same wrong answer consistently for 30 percent of unknown facts, 5-sample recall fell to 68 percent, and with 60 percent to 39 percent.
• Neither detector replaces grounding. The mitigations are structural: retrieve and cite, constrain answers to the context, let the agent abstain ("I do not know"), check numbers and identifiers against tools, and keep a human gate for high-stakes claims.
• Honest limits: the claim set is tiny, templated and hand-written; the consistency model is a probability model; no LLM, entailment model or retriever is used.