Part 1 of 4 · 1 min read
Overview
Agents make many model calls per task, so the temptation is to tune a small model for one step to cut cost and latency. That is sometimes right (a stable, high-volume step with a strict format) and often a costly detour (a step whose failures are missing facts). Knowing which is which saves weeks. Comparing a tuned small model with a prompted baseline is only meaningful if you have the evaluation work from the earlier levels and the cost numbers from Level 11.
About the example, and what is real
Real: the decision procedure, break-even calculator, staleness simulation and sample-efficiency toy, and the 22 tests in code/mini-finetune (plain Python and NumPy). Assumed: every price, volume, upkeep, refresh, accuracy and change rate, and the thresholds in decision.py. Not real: the sample-efficiency experiment uses logistic regression and a one-nearest-neighbour "in-context" stand-in on synthetic data; it shows the shape of data-hungry tuning versus fixed-context prompting, not how a language model behaves. No model was tuned in this module (that starts in module 2).