Hallucination, and two cheap detectors

You ask an assistant for a source and it gives you a paper title, authors and a year. The paper does not exist. The answer looked exactly like a true one. This is the most famous failure.

Why it happens. Module 3 explained that a model is trained to produce likely continuations, not true ones. It has no built-in "I looked this up" check. If the true answer was rare or absent in its training text, a plausible one fills the gap.

What to do:

• Ground the answer. Give the model the source text (retrieval, Level 4) and tell it to answer only from that.
• Ask for evidence. Require a quote or an ID for each claim, then check that the quote exists.
• Check what you can check. Numbers, dates, names, URLs and function names can often be verified by code.
• Let it say "I do not know", and test that it does.

Two cheap detectors

The example code (code/mini-probes) has two small detectors. Both are smoke tests, not guarantees. Real output:

src = ["The invoice total was 1,250 dollars and it was due on 3 March."]
ans = "The invoice total was 1,250 dollars. The customer paid 900 dollars late."

ungrounded_sentences(ans, src)   # ['The customer paid 900 dollars late.']
invented_numbers(ans, src)       # {'900'}

ungrounded_sentences flags answer sentences where fewer than half of the content words appear in any single source. invented_numbers lists numbers that are in no source.

Their blind spots, tried on the same source:

So the first detector misses a changed number and a negation, and it flags honest paraphrases. The second only sees digits. Use several detectors, and for high-stakes claims use a stronger check (a second model asked to verify against the source, or a person).

A bug I found and fixed. The old number detector read a number at the end of a sentence together with the full stop, so 1,250. looked like a different number from 1,250 and a faithful answer was wrongly flagged. It also treated 1250 and 1,250 as different. The function now strips trailing punctuation and thousands separators, and two new tests cover this (10 tests pass).