Accuracy against cost

Accuracy against cost

Plotting all strategies on one chart is the useful habit (log scale on tokens):

Reading it: in this toy, thinking is the efficient first move (the model's steps are what is failing), a good checker is by far the cheapest way to add accuracy when one exists, and voting is the expensive last increment. Change my assumptions (scattered wrong answers, a model that thinks badly) and the order of the first two can swap. The method, not the ranking, is the lesson: measure the curve on your own task.

Choosing the budget per problem

Some models spend a fixed amount of thinking whatever the question. Then a router that guesses how hard a problem is and sets the budget saves tokens:

The routed policies keep the accuracy and cut tokens by about a third, even with a noisy guess, because a one-step error is covered by the slack. (If your provider's model already stops when done, the saving is smaller.)

In an agent

An agent calls the model many times, so a thinking budget is paid per call. Using the toy figures: an agent run with 6 model calls at 250 thinking tokens each adds about 1,500 tokens and several seconds of waiting per task, before any retries. Guidelines:

1. Think where the decision is hard. Planning (module 3), choosing between tools that overlap, reading an ambiguous result, writing a hard query. Skip it for formatting a call, copying fields, or routing between obvious cases.
2. Check where you can. If a test, a validator or a lookup can tell right from wrong (module 4), spend on attempts plus a checker instead of long thinking by itself.
3. Cap everything. A thinking budget, a maximum number of attempts and a cost limit per run (module 6). Tokens are a resource, like steps.
4. Do not stack methods blindly. Reflection (module 4) on top of long thinking on top of voting multiplies the bill; measure each addition.
5. Mind latency. Thinking tokens are generated before the answer, so the user waits for them. Stream status updates, or run attempts in parallel (module 3's rounds).
6. Follow your provider's guidance. Reasoning models are often documented as working best with short, direct prompts and a stated goal, rather than step-by-step instructions written for older models. This is general guidance, so check the current documentation of the model you use.

A checklist

1. Write down the score and the cost (tokens, time) you care about.
2. Start with one attempt, no extra thinking, and measure.
3. Try a thinking budget sweep and plot accuracy against average and worst-case tokens.
4. If a checker exists, try attempts plus the checker before voting.
5. If you vote, look at easy and hard problems separately, and check whether wrong answers repeat.
6. Pick the cheapest setting that meets your target, and re-measure when the model changes.

For the curious Guards added

Voting with zero samples crashed on an empty list, and an evaluation filtered down to no problems divided by zero. Both now raise a clear ValueError. The solver is a probability model, not a language model: each reasoning step succeeds with a fixed chance, written-out steps succeed more often, and the tables show what follows from those assumptions.