Part 1 of 4 · 1 min read
Overview
Agents act, so how they reach an answer matters: an agent that gets the refund right by skipping verification is a security incident that happens to pass the test. Interviewers ask how you evaluate beyond the final answer; the strong answer separates outcome, rules and diagnostics.
About the example, and what is real
Real: the metrics in code/mini-trajectory (exact, in-order, any-order, precision, recall, F1, longest-common-subsequence milestone score, invariant check) and its 11 tests. Assumed: one task (look up an order, verify identity, refund, email), one reference path, one invariant, and five hand-built kinds of run. No agent runs. Because each kind of run is fixed, every pass rate is 0 or 1; the table shows how scorers disagree, not how often real agents make each mistake.