Part 4 of 6 · 3 min read
The gate
Refuse when the evidence is thin.
The gate: refuse when the evidence is thin
The example's abstain.should_abstain applies two rules to the retrieved passages before calling the model:
- Entities must be present. Capitalised names and error codes in the question (for example "Enterprise", "Okta", "E-4021") must appear in the passages. "How much does the Enterprise plan cost?" retrieves the Zephyr and Starter prices, which match "plan" and "cost", but "Enterprise" is nowhere, so it refuses.
- Enough of the question must be covered. If under half the question's content words appear in the passages, refuse.
| System | Answerable correct (18) | Unanswerable correct (6) | Confident wrong answers passed by the checker |
|---|---|---|---|
| answer always | 83% | 67% | 3 of 3 |
| gate before answering | 83% | 100% | 1 of 1 |
The gate fixes the two "Enterprise" questions at no cost on answerable ones in this toy set. Look at what it cannot fix: the Starter extra-seat trap (all its entities are present) still goes wrong, and two answerable questions are refused falsely ("relocate my information to a different area", "get my money back on a yearly subscription") because the keyword retriever finds nothing for those paraphrases. Refusals have two error types, wrongly answering and wrongly refusing, and tightening one loosens the other. Better retrieval (module 3's hybrid search, module 4's query transformations) reduces false refusals; a lower threshold reduces them too but admits more wrong answers.
Practical advice
- Cite by number and resolve in code. Keep a map from
[n]to document ID, title and URL, and render the links yourself. - Make the refusal exact and handle it in code.
- Verify every sentence, not just the answer as a whole, and decide what happens on failure (retry once, strip the sentence, or show a warning).
- Prefer quoting for high-stakes facts (prices, dates, limits) and check quotes verbatim.
- Keep sentences self-contained in your documents so a quoted sentence still names its subject.
- Test refusals as well as answers. Include questions the sources cannot answer, and count false refusals separately.
- Show sources to the user. Even a correct citation is only useful if the reader can open it.
- Remember what you did not check. Meaning, relevance and the truth of the source itself are all outside a citation check.
Interview angle
"How do you make a RAG system trustworthy?" Three layers. Before the model, a gate that refuses when retrieval is thin. In the prompt, numbered sources, "use only these", and an exact refusal sentence. After the model, a verifier that checks each claim against its cited source. Then show the limits: a citation proves what the model says it used, not that the claim is right; a word-matching verifier misses negation; a faithful answer can still be irrelevant. Say you would measure faithfulness and refusal errors separately on a test set that includes unanswerable questions, and use an entailment model or judge for meaning.
Check yourself
An answer sentence says "Monthly plans are refundable [2]", and source 2 says "Monthly plans are not refundable". Does the example checker catch it? (No. It is a word-overlap check, so "not" does not stop the match. You need an entailment model, a judge, or a human for negation.)
For the curious A hole found in the prompt builder
Source text was placed between <<< and >>> as it was. A passage containing >>> could close its own fence and make what follows look like part of the prompt. The builder now breaks up those markers inside source text, and a test checks it. It is still only a speed bump: a model can follow instructions that sit inside a perfectly closed fence (Level 10).
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Numbered sources in the prompt | Every retrieved passage gets an id, and the answer must cite ids. |
| A citation check after generation | Code checks that every cited id exists and the quoted text is in the source. |
| A refusal path | When retrieval is weak, the system says "I could not find this" instead of guessing. |
Draft: confirm whether your model provider has a built-in citations feature before building your own.