Part 4 of 5 · 3 min read
Exercises
Easy
Easy 1: Classify five tasks (20 minutes)
For each, answer the six questions and run recommend: (a) turn invoices into a spreadsheet, (b) answer customer questions from a policy document, (c) investigate a failing deployment, (d) translate a document, (e) plan a week of travel with bookings.
Done when: five verdicts, plus one case where you disagree with the tool and why.
Easy 2: Name the cheaper option (15 minutes)
For three tasks of your own, say what non-model solution (rule, query, lookup) could work, and what would force you to use a model.
Done when: three tasks, three alternatives, three reasons.
Easy 3: Read the tail (15 minutes)
From simulate, find the mean and 95th percentile calls for the agent at 30% novelty. If each call costs 1 cent, what should you budget per ticket to be safe, and why is the mean not enough?
Done when: you give both numbers and a sentence of reasoning.
Hint
budget for the 95th percentile, and set a hard cap at the maximum, so one runaway ticket cannot dominate the bill.
Medium
Medium 1: Change the assumptions (60 minutes)
Re-run the simulation with p_step at 0.90 and 0.99, and with max_steps of 6 and 20. Tabulate success and calls. Which parameter matters most, and what does that tell you about where to invest effort?
Done when: a table of four or more runs and a conclusion.
Hint
vary one parameter at a time and keep the seed fixed.
Medium 2: Add an escalation path (60 minutes)
Change the workflow so a novel ticket is escalated to a human at a fixed extra cost instead of counting as a failure. Compare the cost per solved ticket for both systems when a human costs 50 times a model call.
Done when: a plot or table shows where the agent becomes the cheaper choice, if it does.
Hint
cost per solved ticket is total cost divided by solved tickets, including human cost.
Medium 3: Add step zero (60 minutes)
Add a seventh question to recommend: "Is the input structured and the rule simple?" with a result of "no model needed". Update the tests and explain how the question changes the leaning.
Done when: tests cover the new outcome and an old test still passes.
Hard
Hard 1: Hybrid simulation (120 minutes)
Simulate a three-stage pipeline where only stage two is an agent (with a cap of 6 steps) and stages one and three are single calls with a code check after the agent. Compare success, mean and 95th percentile calls with a pure workflow and a pure agent.
Done when: a chart shows all three and you state where the hybrid wins and loses.
Hint
reuse run_agent with agent_base_steps=3 only for the middle stage.
Hard 2: Measure for real (120 minutes)
Pick a task you can test with 30 labelled examples. Build a workflow version and a simple agent version (Level 2, module 4's loop) against a real model, and compare success rate, mean and worst-case calls, and cost. (I did not run this, so your numbers are the ground truth.)
Done when: a results table and a one-page recommendation.
Hard 3: The written argument (120 minutes)
Write the one-page argument for why the support-triage flow from module 3 is a workflow and not an agent, using real numbers from your own tests (or the simulation, clearly labelled). Include what evidence would change your mind.
Done when: a peer or reviewer can find the four elements (what varies, what does not, cost of an error, trigger for an agent stage).