Why models make things up

Why models make things up

Two explanations are discussed in the literature, and they suggest different cures:

1. Self-delusion (snowballing). A model treats text it generated earlier the same way as facts it was given. After a slightly wrong first sentence, later tokens are conditioned on it and can pile detail onto the mistake. A wrong first step can make the model fail on things it would otherwise know. Cure direction: separate what was given from what was generated; keep answers short and check early.
2. Knowledge mismatch. If training examples use knowledge the model does not have, the model learns to sound knowledgeable anyway. Cure direction: make the model answer only from what it can show (sources), and punish made-up claims.

Retrieval supplies facts at the moment of answering, but the model is still free to ignore them, mix them with memory, be distracted by irrelevant passages, or answer when the passages say nothing. So the work moves from "supply facts" to "make sure the facts are used, and say so when they are missing".

Four places to act

2.1 Choose the context

More passages are not better. Extra passages that look relevant but are not (a different plan's price, an older policy) compete with the right one. In the example, with temperature 1 the share of confident wrong answers rises from 29 percent with one passage to 40 percent with five or eight. Where passages sit also matters, because models have been reported to use the start and the end of a long prompt better than the middle.

Order the context so the best passage is first, or put the best at the start and the next best at the end. In the example (with the penalty on), ordering by relevance raises "correct" from 51 percent (documents in arbitrary id order) to 61 percent. Because the dip is an assumption in this toy, treat that as an illustration of the method; test your own model with a "needle in a haystack" check (insert a known fact at different positions and see whether it is found).

2.2 Constrain the model

Module 5's prompt rules (use only these sources, cite as [n], an exact refusal sentence) belong here. So does lowering the temperature for factual tasks: in the example, going from 1.5 to 0.25 cuts wrong answers from 41 to 19 percent. Low temperature makes the model repeat its best guess, which helps only if its best guess is good, so it is not a cure on its own.

2.3 Check the answer

Module 5's per-sentence checker catches misquotes (a changed number, an invented detail). The example imitates a model that misquotes 25 percent of the time it answers. Without a check, about 10 percent of all answers fail the source check; with check-and-retry (up to three tries, then refuse), none do, and the confident wrong share falls from 31 to 23 percent. It does not fall to zero because many wrong answers are faithful but irrelevant: they quote a real sentence that does not answer the question (module 5, section 4).

2.4 Decide whether to answer: self-consistency

If a model "knows" something, repeated samples agree. If it is guessing, they scatter. The example samples five answers per question and refuses unless a fraction tau of them match:

This is a risk and coverage curve. Every point on it is a legitimate setting; the right one depends on how costly a wrong answer is compared with a refusal. A medical or legal assistant sits far down the table; a casual helper sits near the top. The price is real: five samples cost five times the model calls and add delay.