Part 5 of 6 · 3 min read
Exercises
Easy
Easy 1: Spot the bad example (15 minutes)
A prompt has these examples for labels billing, bug: ("it crashes", bug), ("it crashes", billing), ("charge twice", defect). List every problem and fix each.
Done when: you name three problems and give a corrected set.
Hint
a duplicate input with conflicting answers, a label not in the set, and an unbalanced or too-small set.
Easy 2: Hand-compute voting (20 minutes)
With p = 0.6, compute by hand the chance that a vote of 3 is right, then check with expected_vote_accuracy. Then explain why voting with p = 0.4 makes things worse.
Done when: both numbers agree and the second claim is demonstrated with code.
Hint
for p = 0.6, 3p^2(1-p) + p^3 = 0.648.
Easy 3: Write the answer contract (15 minutes)
Write the reasoning instructions (as text) for a prompt that must return a total price. Include the line format of steps and the last line.
Done when: parse_answer and steps_of work on a reply you invent that follows your contract.
Medium
Medium 1: Example count curve (60 minutes)
Using the imitation model, measure accuracy with 0, 1, 2, 3 and 5 on-topic examples (add more to the pool if you need). Plot or tabulate it. Explain why the curve flattens for this model, and say why a real model's curve might differ.
Done when: you have a table and two honest paragraphs.
Medium 2: Verify more (60 minutes)
Extend verify_arithmetic to handle percentages such as "20% of 50 = 10", powers such as "2 ** 3 = 8", and amounts with a currency sign such as "$12 * 3 = $36". Test each new form, and test that a correct line is never flagged.
Done when: at least five new tests pass.
Hint
strip $ and , first, turn ** into a power in the evaluator, and treat "N% of M" as N / 100 * M.
Medium 3: Leakage detector (60 minutes)
Extend check_examples to flag test inputs that are near an example (word overlap above 0.8), not just identical. Test with a reworded duplicate.
Done when: it flags a reworded leak and does not flag unrelated tickets.
Hint
reuse words from fakemodel.py and a Jaccard overlap (shared words divided by all words).
Hard
Hard 1: Measure on a real model (120 minutes)
Using a real model, compare zero-shot, three fixed examples and three selected examples on 30 tickets you label yourself, with a separate test set. Report accuracy, tokens per request (Level 2, module 6) and cost per correct answer. (I did not run this, so your numbers are the ground truth.)
Done when: you have a table and a recommendation with its caveats.
Hard 2: Correlated samples (120 minutes)
Change NoisySolver so each question has a hidden "trap" that makes all samples slip the same way with some probability. Show that the vote's accuracy now plateaus well below the independent case, and explain what that means for choosing n.
Done when: a plot or table compares independent and correlated cases.
Hint
draw the trap once per question, outside the sampling loop.
Hard 3: Verify and retry (120 minutes)
Build a loop that asks the solver, runs verify_arithmetic, and on a failure asks again with the wrong line quoted back, up to 3 times (the same idea as Level 2, module 3). Measure how often the final answer is correct for p = 0.5, 0.7, 0.9.
Done when: a table shows accuracy against the no-verification baseline.