Part 5 of 6 · 3 min read
Exercises
Write each answer in your own words and code. Use the placeholder prices unless you have a real price list.
Easy
Easy 1: Price by hand (15 minutes)
A call has 4,000 input tokens and 300 output tokens on the "small" model. Work out the cost on paper, then check with cost_of.
Done when: both numbers match.
Hint
the small prices are 0.25 in and 1.25 out per million.
Easy 2: Spot the cache breaker (15 minutes)
List five things that could appear near the top of a prompt and would stop prefix caching from working. For each, say where it should go instead.
Done when: you have five items with a fix each.
Easy 3: Re-price (20 minutes)
Load a saved ledger and compute what the same usage would cost if the "large" input price doubled. Explain why storing tokens made this possible.
Done when: you print the old and new totals.
Hint
Ledger.load keeps the stored dollars. Re-price by calling cost_of(record.usage, new_price) on each record.
Medium
Medium 1: Cost report (60 minutes)
Write a function that reads a ledger file and prints cost per user, per model and per day, plus the single most expensive run and its step count. (Add a timestamp field to the record for the per-day view.)
Done when: it works on a ledger of at least 200 generated records, with a test.
Hint
collections.defaultdict(float) and the record's run_id.
Medium 2: Warn before you stop (60 minutes)
Extend BudgetGuard with a soft limit at 80% that logs a warning once, and a hard limit that raises. Test both, and test that the warning appears only once.
Done when: three tests pass.
Hint
keep a boolean warned on the guard.
Medium 3: Which layout wins? (60 minutes)
Using PrefixCache, compare three layouts of the same content (stable first, stable last, history before document). Report hit share and total cost for 30 requests each.
Done when: a table of three rows is printed and you explain the order.
Hard
Hard 1: Instrument the agent loop (120 minutes)
Wrap your module 4 run_with_tools so every model call is logged to the ledger with run ID and step, retries included, and the guard is checked before each call with estimate_call. Show a run stopped by a $0.02 cap.
Done when: a test shows the stop and the ledger total is at or below the cap.
Hard 2: Break-even for caching (120 minutes)
Some providers charge more to write a cache entry than to read it. With the "large" placeholder prices, work out how many reuses of a 10,000-token prefix are needed before caching is cheaper than not caching, then confirm with the simulation. Repeat with a 5-minute expiry and requests every 4 and every 6 minutes.
Done when: you give the formula, the numbers, and a simulation that agrees.
Hint
caching wins when write_cost + n x hit_cost < (n + 1) x normal_cost. Solve for n. With a 5-minute expiry, requests every 6 minutes never find a live entry.
Hard 3: Reconcile with a fake invoice (120 minutes)
Generate a fake provider invoice that includes 3% more calls than your ledger (simulated retries you forgot to log). Build a tool that finds the missing share by comparing counts per hour and proposes which part of the code is not logging.
Done when: the tool flags the gap, and the test shows the fix closes it to under 0.5%.
Hint
bucket records by hour and compare counts. The hour with a persistent excess points at the missing path.