Covers: Results; How to set a goal an agent can be held to.
Results
Setup: five subgoals; 15 percent of tasks cannot get past subgoal 3; budget of 30 steps where one applies, otherwise a hard cap of 100; 20,000 tasks per policy, seed 0.
Policy
Finished
Said done, was not
Escalated rightly
Escalated wrongly
Stopped by budget or cap
Mean cost
Wasted cost
Cost per success
agent says done
59%
40%
0
0
0.5%
21.8
10.1
36.8
agent says done + budget
49%
30%
0
0
21%
17.8
10.8
36.3
verified + budget
64%
0.9%
0
0
35%
20.3
10.6
31.7
verified + stall detector
77%
0.8%
15%
7.5%
0
21.8
6.7
28.3
verified + stall + budget
76%
0.8%
13%
7.0%
3.5%
21.6
7.0
28.4
Outcomes under five policies, false completions against the over-claim rate, and the effect of the stall threshold.
False completions as the agent over-claims more (agent says done): 0 percent at no over-claiming, 25 percent at 0.02 per step, 41 percent at 0.05, 55 percent at 0.10.
A leaky verifier (verified + stall + budget): false completions 0 at a verifier error of 0, 0.7 percent at 2 percent, 3.4 percent at 10 percent, 9.6 percent at 30 percent. A weak verifier is a weak goal: it is only as good as the criteria it checks.
The stall threshold (verified + stall + budget):
Steps without progress before a replan
2
3
5
8
12
Escalated although the task was possible
30%
15%
7%
2%
0
Escalated impossible tasks
15%
15%
13%
6%
0
Mean cost
14.2
17.6
21.5
24.5
25.7
Correct outcomes
0.70
0.84
0.89
0.81
0.71
The share of impossible tasks (correct outcomes, verified + budget against verified + stall + budget): 0 percent: 0.75 against 0.89; 15 percent: 0.64 against 0.89; 40 percent: 0.45 against 0.88.
Reading it:
Self-report is the weakest stop signal. It is free, but it produces a wrong "done" in two of every five tasks when the agent over-claims at 5 percent per step. The rate is my assumption; the structure (a claim is made by the same system that may be wrong) is not.
A verifier turns a claim into a decision. It needs success criteria you can check; that is the reason "make the goal checkable" comes first.
A budget alone converts every hard task into a timeout. It bounds cost but gives no information: 35 percent of tasks ended there, some of them impossible and some just slow.
A stall detector pays for itself even though it costs per step. Mean cost per task rose from 20.3 to 21.8 but wasted cost fell from 10.6 to 6.7 and cost per success fell from 31.7 to 28.3.
The threshold has a sweet spot. Too small a threshold makes normal variation look like a stall (the agent needs about two steps per subgoal, so five idle steps in a row is unusual but not rare); too large a threshold means the budget ends the run before the monitor acts.
Escalation is a feature, not a failure. 15 percent of tasks were impossible, and "escalated rightly" is the best possible outcome for them.
How to set a goal an agent can be held to
Write the success criteria as checks. "Report includes three sourced findings" and "tests pass" are checkable; "a good report" is not, unless a rubric and a judge are part of the design.
Separate the claimer from the checker. A different model, a test suite or a person; never the same prompt grading itself.
Set limits per layer: steps per task, tokens, wall-clock time, and number of replans.
Define the escalation path (who, with what context) before you need it, and test it with an impossible task.
Log a progress ledger per step (subgoals done, steps idle, spend), so monitoring is also your trace (Level 9).
Use a cheaper model for monitoring when possible, since it runs on every step; the simulation charged 0.3 model calls per step.
Keep a budget even with a monitor, since monitors fail too.