Definitions

Definitions

The pipeline

Results

1. Reading more helped only where the first retrieval missed; it also made conflicts visible.
2. A verifier turned an unreliable writer into a correct report because it checked claims against pages, not against the writer's confidence.
3. The cost is real: 39 pages instead of 24, before any model tokens are counted.

Design rules

• Plan the needs explicitly and track which are covered.
• Search per need, not once per question, and stop when the need is covered by a primary source.
• Rank sources by kind and recency, record dates, and display conflicts instead of hiding them.
• Verify every citation mechanically where possible; sample by hand otherwise.
• Say "not found" rather than filling a gap; report the gaps.
• Quarantine instruction-like page text; keep research-phase tools read-only.
• Check origin, not count: three pages repeating one source are one source.
• Budget sources, steps, time and money; checkpoint long runs.
• Show the trail: a source list with dates and the claims each supports, so a reader can check in seconds.
• Evaluate with traps: stale sources, conflicts, missing facts, and planted instructions.

Interview questions, with the follow-ups

Common mistakes

• One retrieval for a multi-part question, then confident synthesis.
• Citing a page that does not contain the claim.
• Hiding conflicts to make the answer look clean.
• Trusting forum posts or very old pages without dates.
• Treating page text as instructions.
• Counting repeated claims as independent confirmation.
• No "not found" path, so the agent invents.
• No budget, so one hard question burns the day's spend.
• Evaluating only the prose, not the claims and sources.
• Dropping source dates and URLs from the report.

What the example does not do

• No model and no web: the planner is rules, the writer is a script and the corpus is invented.
• Claims are structured (CLAIM entity | attribute | value), which makes verification exact; real pages need extraction, matching and judgement, which are the hard part.
• The corpus is tiny, so retrieval is easy; real corpora have duplicates, near-duplicates, paywalls and noise.
• The injection defence is a regular expression, easy to evade; it demonstrates the structure (data, not instructions), not a robust filter.
• No citation-laundering detection, no source-origin tracing, no handling of numbers with units or ranges.
• The ranking rule (kind, then date) is a simple choice; recency can be wrong when an old primary source is the origin.
• Optional real experiment (I did not run this): pick ten factual questions with known answers in a domain you know, run a real model with search, and score coverage, correctness, citation precision (open each cited page) and cost; then add a verifier and repeat.

For the curious Citations need a check

A careless writer, simulated with 25 percent of statements getting a wrong citation or a changed value, produced 4.6 bad citations on average (14.6 of 17 correct) over 30 seeds. A verifier that checks whether the cited page contains the claim, then repairs or regenerates it, gave 0 bad citations and 17 of 17 correct. That check is mechanical on my invented corpus; real pages need text matching or an entailment check and a human spot check.

What you must be able to say (one page)

• A research agent turns a question into a sourced answer. It gathers, synthesises and summarises information from documents, knowledge bases or the web into a structured output for a human analyst; Perplexity and Elicit are products in this area. A "deep research" agent tackles open-ended investigations that need extensive outside knowledge gathering, hypothesis testing and synthesis. The price is high compute cost and very high latency, and the best fit is long-form literature review.

• The pipeline: plan the needs (which facts does the question require?), search with a budget, read pages as untrusted data, resolve conflicts between sources, write each claim with a source id (or "not found"), then verify that each cited page really says it. Plan first so you know when to stop and what is still missing.

• Per-need search beats one retrieval for the whole question. In my eight-question experiment (invented corpus of 16 pages, 17 needed facts): a single top-3 retrieval answered 14 of 17 and got 13 right, reading 24 pages; per-need search answered 17 of 17 correctly, reading 39 pages. The single-shot run reported a latency of 15 ms taken from a forum post because the primary page (42 ms) was never retrieved, so no conflict was even visible. Retrieval depth is a cost-versus-coverage dial: with a budget of 3 sources per question only 12 of 17 were answered; 6 was enough here.
• Sources disagree, so show it. Rank by source kind (primary, secondary, forum) and then recency, but report the chosen value, its source, and the values that disagree. In my run, six of the 17 answers carried a visible disagreement (an old blog said 80 ms and a forum post said 15 ms against the official 42 ms; a forum rumour said GPL-3 against the official BSD-3). Flag answers that rest on low-trust sources only.
• Citations must be checked, not trusted. I simulated a careless writer (25 percent of statements get a wrong citation or a changed value, a stand-in for model errors). Without a verifier the report had 4.6 bad citations on average (14.6 of 17 correct) over 30 seeds; with a citation verifier (does the cited page contain this claim? repair, else regenerate from the evidence actually read) it had 0 bad citations and 17 of 17 correct. This is a mechanical check against structured claims in my corpus; real pages need text-matching or entailment checks (Level 9 module 6) and a human spot check.
• Web content is untrusted input. One page in my corpus contained an instruction aimed at the agent; the reader drops instruction-like sentences, marks the page suspect and lowers its trust, and answers citing suspect pages are flagged (Level 10 module 1). Sanitising is not a complete defence: keep tools minimal during research and never let page text change the plan.
• Know the weak points: stale sources, SEO spam, paywalls and missing pages, confident synthesis over thin evidence, citation laundering (a claim copied across sources that trace to one origin), long runs that need checkpoints and cancellation (Level 11 module 3), and cost (one run in my Level 11 estimate was about $7).
• Evaluate with questions that have gold answers and known traps: coverage, correctness, citation precision, conflict handling, refusal to answer when evidence is missing, and cost and latency. Outcome and trajectory evals both matter (Level 9 module 3).
• Honest limits: an invented 16-page corpus with structured claims, a rule-based planner and a scripted careless writer. It tests the pipeline's logic, not a model's reading comprehension or the real web.