Part 4 of 5 · 4 min read
Exercises
Easy
Easy 1: Run and read (20 minutes)
In code/mini-think, run pytest and python demo.py. From sections 2 and 3, report accuracy and average tokens at thinking budgets 0 and 480, and the accuracy of voting with 1, 9 and 33 attempts on hard problems.
Done when: you give both tables and explain in two sentences why voting got worse on hard problems.
Hint
on hard problems the right answer is rarer than the attractive wrong one.
Easy 2: Compute by hand (15 minutes)
Using the rules (written step 93%, head step 75%, 60 tokens per written step), compute the chance of a correct answer and the tokens used for a 4-step problem with budgets 0, 120 and 480.
Done when: you have six numbers and check the 120-token case against the demo's first table (49%).
Hint
with budget 120, two steps are written (93%) and two are in the head (75%): 0.93 x 0.93 x 0.75 x 0.75.
Easy 3: Which technique? (15 minutes)
For each, say whether you would use more thinking, voting, a checker, or none, and why: (a) a classification with three clear labels; (b) a SQL query with a test database to run it on; (c) a multi-step maths word problem; (d) choosing between two search tools.
Done when: each answer names a technique and one reason that mentions steps, errors or checkability.
Hint
ask first whether anything can check the answer.
Medium
Medium 1: Scattered or clustered (60 minutes)
Sweep the attractive-wrong share from 0 to 0.8 in steps of 0.2. For each, report the accuracy of voting with 9 attempts and at what share voting stops beating a single attempt.
Done when: you have a table and a stated crossover point, and one sentence on how you would measure this share for a real model.
Hint
the crossover is where the right answer's share per attempt equals the attractive wrong answer's share.
Medium 2: Weighted votes (60 minutes)
Implement a vote where each attempt is weighted by a noisy checker's score (right answers score higher on average). Compare unweighted voting, weighted voting and first-accepted at 9 attempts.
Done when: you report three accuracies and the cost of each, and say which assumption about the checker the weighted vote needs.
Hint
weights can come from tpr and fpr; do not let a weight dominate on its own.
Medium 3: Make the router smarter (90 minutes)
Improve the budget router: use the first attempt's answer length or a second quick attempt to decide whether to think more. Compare with the fixed budget and the noisy-guess router on accuracy and tokens.
Done when: you show your router on the accuracy and tokens chart next to the fixed policies and say whether it is on the efficient frontier.
Hint
a cheap second attempt that disagrees with the first is a signal of difficulty.
Hard
Hard 1: A real thinking sweep (150 minutes)
Pick 40 questions of varied difficulty with checkable answers. With a real reasoning model, sweep the thinking budget (or reasoning effort setting) over at least four levels and record accuracy, average tokens and 95th percentile tokens. (I did not run this, so your numbers are the ground truth.)
Done when: you have a table and a chart of accuracy against tokens, and say where returns diminish.
Hint
fix the question set and temperature before the sweep; keep the raw answers.
Hard 2: Do wrong answers repeat? (150 minutes)
For 40 questions, sample a real model 10 times at some temperature. Count, per question, the share of wrong answers that are the same wrong answer, and predict whether voting will help. Then run voting and compare with your prediction.
Done when: you report the clustering measure, the prediction, the result, and one question where voting hurt (if any).
Hint
compare answers after normalising (strip spaces, lower-case numbers); count ties carefully.
Hard 3: Thinking inside an agent (180 minutes)
Take your agent from Level 5 and give each step type its own thinking setting (planning, tool choice, final answer). Run 20 tasks across at least three settings and compare success, tokens, wall-clock time and cost per success.
Done when: a table shows which step type benefits most from thinking, and the cheapest setting that keeps success within 2 points of the best is named.
Hint
change one step type at a time before trying combinations.