Check each sentence against its source, and why a cited answer can still be wrong.
Checking each sentence against its source
Citation check
For every sentence in the answer, test that (a) it has a citation, (b) the citation points to a source that was really in the prompt, and (c) the cited source actually contains the claim.
The example's verify.check returns one status per sentence:
Status
Meaning
ok
all numbers and at least 75% of the content words appear in the cited source(s)
uncited
no [n] marker
bad-ref
the marker names a source that was not in the prompt (for example [9])
unsupported
the cited source does not contain the claim
check("The Zephyr plan costs 45 dollars per month [1].",["p1"])[0].status# 'unsupported' (the source says 49)check("The Zephyr plan costs 49 dollars per month [4].",["p1"])[0].status# 'bad-ref'check("The Zephyr plan costs 49 dollars per month.",["p1"])[0].status# 'uncited'
There is also a stricter tool for quotations: quote_found(quote, source) requires the quoted words to appear in the source verbatim, ignoring case and spacing. If you ask the model to quote, you can check mechanically.
What the checker catches and misses
To test the checker, the example takes honest answers and damages them on purpose in seven ways, then counts how many damaged answers get flagged.
Share of damaged answers the checker flagged (toy checker; honest answers damaged on purpose)
It catches every changed number, wrong source, made-up reference, deleted citation, uncited extra claim and invented detail. It misses both "negation flips" (removing "not" from "Monthly plans are not refundable"), because every word of "Monthly plans are refundable" is still in the source. That is the central limit of word-matching: it checks that the words are present, not what the sentence means. A real system adds an entailment model (does the source imply the claim?) or an LLM judge, and tests it as carefully as any other component (module 7 of this level and Level 9).
A cited answer can still be wrong
Citation checking answers one question: "does the cited source contain this sentence?" It does not ask "does this sentence answer the user's question?" The example has a case that shows the gap:
Q: What is the monthly price of an extra seat on the Starter plan?
A: On the Zephyr plan each extra seat is billed at 9 dollars per month [1].
The sentence is true, cited correctly, and passes the checker, but it is about the wrong plan (the Starter document says seats cannot be added). In the run, the confident wrong answers all pass the checker: three of three without a gate, and one of one with it. Faithfulness (the answer matches its sources) and relevance or correctness (the answer matches the question) are separate properties and need separate checks. Reference-answer evaluation (Level 9) is the usual remedy.