Covers: Definitions; Five questions to ask and more.
Definitions
Benchmark, leaderboard, harness
Benchmark: a fixed set of tasks and a scoring rule used to compare systems.
Leaderboard: a ranked list of reported scores.
Harness (scaffold): the code, prompts and tools wrapped around a model when it is run on the benchmark.
Saturation, contamination, ceiling
Saturation: scores approach the maximum, so differences vanish.
Contamination: benchmark items or their solutions appear in a model's training data.
Ceiling: the highest achievable score, below 100 percent if some tasks are flawed.
Paired bootstrap, power
Paired bootstrap: resampling tasks, keeping both agents' result for each task, to see how often the difference changes sign.
Power: the chance of detecting a real difference.
Five questions to ask
What is measured, how big, how clean, how contaminated, which harness.
Benchmark
Measures
What it can miss (my assessment)
GAIA
multi-step questions with web, files and tools; exact-match answers
long, open-ended work; actions with side effects; cost and latency; answers may be findable after publication
SWE-bench family
resolving real repository issues so that tests pass
test flaws, contamination (OpenAI's audit), no measure of code review quality or maintainability
tau-bench (general knowledge)
dialogue with tools under policy rules, repeated for consistency
domain coverage; simulated users
The right-hand column is my judgment, partly from the sources above, not a published finding for each row.
Results
Ranking five agents of skill 0.50, 0.53, 0.56, 0.59 and 0.62 on a benchmark of n tasks (600 repeats):
Tasks
50
100
200
500
1,000
Best agent ranked first
54%
63%
76%
88%
96%
Detecting a true gap (agent A against agent B, paired bootstrap, 200 repeats): 4-point gap: 22 percent at 100 tasks, 60 at 500; 2-point gap: 11 at 100, 26 at 500.
Flawed tasks (500 to 4,000 tasks; skills 0.62 against 0.50): 0 percent flawed: gap 12.5 points, ceiling 100; 20 percent: 9.7, 80; 40 percent: 7.8, 60.
Contamination (a stronger uncontaminated agent scores 64 percent):
Share of the set a weaker agent memorised
0%
10%
20%
30%
40%
Weaker agent's score
45%
50%
54%
61%
67%
Ranking stability, contamination and flawed-task effects.
Treat differences under a few points on a few hundred tasks as noise.
A flawed benchmark compresses what it can show, so progress looks slower than it is, or models look closer than they are.
A contaminated set rewards memory and can reverse the true order.
Public suites are built for models in a standard harness; your agent's tools, data and rules differ.
Design rules
Use public benchmarks to shortlist, then evaluate finalists on your own eval set with intervals.
Check the date and the version: benchmarks are retired or superseded (the SWE-bench Verified case).
Read the harness, the budget and the number of attempts before comparing two rows.
Look for a held-out or private split, and prefer benchmarks with fresh tasks.
Report cost and latency with the score.
Do not tune on the benchmark and then quote it.
Re-run what you can, from the benchmark's own harness, with repeated trials (module 1).
Interview questions, with the follow-ups
How would you use public benchmarks when choosing a model for an agent?
"To shortlist, not to decide. I check what the benchmark measures against my task, its size and noise, whether it is saturated or contaminated, and whether the rows used the same harness and budget. Then I evaluate the finalists on my own traces-based set with intervals and cost. A recent example is SWE-bench Verified, which OpenAI stopped reporting after finding flawed tests and contamination." Follow-up: "What is wrong with a leaderboard rank?" It hides error bars (on 100 tasks a 4-point gap is detected about a fifth of the time in my simulation), harness differences and cost.
What does GAIA measure?
"Real-world assistant questions, 466 of them, that need reasoning, browsing, multi-modality and tool use; they are simple for humans, who scored 92 percent in the paper, and were hard for the model with plugins, 15 percent. It scores final answers by exact match, so it says little about side-effecting actions, cost or long-running work." Follow-up: "Why do you use it in your own project?" As an agent gym: it is a convenient set of multi-step tasks with answers (Build an AI Agent from Scratch), but it is not a proxy for my product.
What is benchmark contamination and how does it matter?
"The model saw the test data or solutions during training, so the score measures memory. In my simulation a weaker agent that had memorised 40 percent of the set outscored a stronger clean agent, 67 to 64.
Two agents differ by 3 points on a 300-task benchmark. Which is better?
"I cannot tell from that alone. With a 3-point gap and 300 tasks the detection power is below one half in my simulation, and a paired comparison on the same tasks with repeated runs is needed. I also check harness and cost." Follow-up: "What would convince you?" A paired test whose interval excludes zero on a task set that resembles mine.
Common mistakes
Choosing a model on a leaderboard rank alone.
Comparing rows with different scaffolds and budgets.
Ignoring the benchmark's date and status.
Quoting a benchmark you tuned on.
Reading small gaps as meaningful.
Using exact-match benchmarks to judge action-taking agents.
No cost column.
What the example does not do
No real benchmark data; difficulty and skill are drawn from simple distributions.
Flaws are all-or-nothing; real flaws also let wrong solutions pass.
Contamination is a fixed share of fully solved items.
One agent run per task; repeated trials are in module 1.
The facts about SWE-bench Verified are OpenAI's own account, as reported on its page; I did not verify the audit.
Optional real experiment (I did not run this): run two models on 100 GAIA validation questions with your harness, three times each, and compute the paired interval for their difference.
Check yourself
A model tops a public leaderboard. What does that tell you about your product? (Only that it is worth shortlisting. Decide on your own eval set, and check the harness, budget and attempts before comparing rows.)
For the curious Benchmarks retire
OpenAI said on 23 February 2026 that it had stopped reporting SWE-bench Verified; its audit of 138 problems found 59.4 percent had flawed tests or unclear descriptions (read 31 October 2026). Check the date and version of any benchmark you quote, and re-check current claims before relying on them.
What you will use daily at work
Practice
What it looks like
Shortlist with public scores
Then test finalists on your own cases.
Read the harness
Tools, retries, time and cost budgets differ between rows.
Report cost next to score
A higher score at ten times the cost may not be better.
Check whether the benchmark is still current
Tasks get retired or superseded.
Draft: confirm each benchmark you quote against its current page before the lesson goes live.
What you must be able to say (one page)
A public benchmark is a hint about general ability, not a measurement of your product. Use it to shortlist models and agents; decide on your own eval set (module 2).
GAIA (Mialon et al., 2023) has 466 questions that need reasoning, multi-modality handling, web browsing and tool use; they are "conceptually simple for humans but hard for advanced AI". The original paper reported 92 percent for humans and 15 percent for GPT-4 with plugins (abstract, read 31 October 2026).
Benchmarks saturate and decay. A benchmark is saturated once models reach the top score; GLUE was saturated within a year. Contamination, models having seen the test data, makes scores less trustworthy.
A current example (read 31 October 2026): on 23 February 2026 OpenAI said it had stopped reporting SWE-bench Verified. Its audit of 138 problems that its o3 model did not consistently solve found that 59.4 percent had flawed tests or unclear descriptions, and that every frontier model it tested could reproduce the original fix or verbatim problem details, which it took as contamination. Scores had risen only from 74.9 to 80.9 percent in six months. It recommended reporting the public split of SWE-bench Pro until uncontaminated replacements exist, while calling that imperfect. This is OpenAI's account; I did not check the audit independently.
In my simulation (assumed difficulty):
Small benchmarks rank unreliably. With five agents 3 points apart, the best one was ranked first 54 percent of the time on 50 tasks, 76 on 200, 88 on 500 and 96 on 1,000.
Power to see a real gap is low. A true 4-point gap was detected (paired bootstrap, 5 percent level) 22 percent of the time on 100 tasks and 60 percent on 500; a 2-point gap, 11 and 26 percent.
Flawed tasks compress differences. With 0, 20 and 40 percent of tasks unpassable, the score gap between a strong and a weak agent fell from 12.5 to 9.7 to 7.8 points, and the ceiling fell to 80 and 60 percent.
Contamination can invert a ranking. An agent with skill 0.45 that had memorised 40 percent of the set scored 67 percent, above an uncontaminated stronger agent's 64 percent.
Rows are not comparable unless the harness is. Scaffolding, tools, retries, time and cost budgets differ between leaderboard rows; compare only like with like, and report cost next to score.
General knowledge, not re-checked today: tau-bench (Yao et al., 2024) scores tool-using customer-service agents and popularised repeating each task to measure consistency (the idea behind pass^k in module 1).