Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-cost.
Done when: you report cost per resolved task for the four policies.
Easy 2: Your prices (30 minutes)
Replace PRICE with your provider's current prices and rerun.
Done when: you note which conclusion changed.
Hint
only the dictionary values change.
Easy 3: Cost of a step (30 minutes)
Compute the cost of your own agent's average run from its token counts.
Done when: you split it into system, history, tool output and output tokens.
Medium
Medium 1: Real cost dashboard (4 hours)
Add token and cost attributes to traces and chart cost per run and per resolved task.
Done when: a dashboard shows both for the last 100 runs.
Hint
record input, cached input and output tokens separately.
Medium 2: Tune the cascade (3 hours)
Sweep verifier accuracy and escalation rules.
Done when: you find where the cascade stops beating large-only.
Hint
the cascade's cost is small + verifier + P(escalate) x large.
Medium 3: Budget behaviour (3 hours)
Make an agent return a partial result and an escalation note when the cap is hit.
Done when: a looping task ends in that state, with the spend logged.
Hard
Hard 1: Replay experiment (8 hours)
Run the real-model replay from section 7. (I did not run this, so your numbers are the ground truth.)
Done when: you report a table like section 3 from your own logs.
Hard 2: Prefix-cache audit (6 hours)
Measure the real cache hit rate of your agent and fix prompt ordering. (I did not run this.)
Done when: hit rate and cost per run are reported before and after.
Hint
cache only helps if the prefix is byte-identical, including whitespace and tool order.
Hard 3: Semantic cache safety (6 hours)
Build a semantic cache on real queries and label wrong hits. (I did not run this.)
Done when: you choose a threshold from the measured wrong-answer rate.