Part 1 of 4 · 1 min read
Overview
Interviewers ask "how would you pick a model for this agent?" and "what do these benchmarks tell you?". The strong answer is: they narrow the field, but their error bars, flaws, contamination and mismatched harnesses mean you must evaluate on your own tasks.
About the example, and what is real
Real: the ranking-stability, paired-bootstrap, flawed-task and contamination simulations in code/mini-bench, with 9 tests. Assumed: task difficulty is Beta(2,2); an agent's chance of solving a task is a simple linear rule in skill minus difficulty; flawed tasks are unpassable; memorised items are always solved. No real benchmark or agent is run. The facts about GAIA and SWE-bench come from the sources named; the simulation numbers are illustrations.