Part 5 of 6 · 3 min read
Exercises
Easy
Easy 1: Count the calls (15 minutes)
For a ticket with two issues, how many calls does orchestrate use? For a loop that passes on round 3 with a code evaluator? For a chain of 4 steps? Predict, then verify with Counted.
Done when: three predictions match three measurements.
Hint
the orchestrator uses 1 + subtasks + 1. The loop uses one generate call per round.
Easy 2: Pick the pattern (20 minutes)
For each task, name the best pattern and give one sentence of reason: (a) translate then summarise a contract, (b) answer questions about billing or about installation using different manuals, (c) check a post for spam, rudeness and length, (d) fix all compiler errors in a repository, (e) polish an email until it has a greeting, a deadline and no jargon.
Done when: five answers, each with a reason.
Easy 3: Draw it (20 minutes)
Draw each pattern for a task of your own choice (pencil is fine) and mark where code, not the model, makes a decision.
Done when: five small diagrams with the decision points circled.
Medium
Medium 1: Retry inside chain (60 minutes)
Make chain retry a failed gate once with the gate's reason added to the prompt before giving up. Test that a flaky step succeeds on the retry and that a hopeless one still stops.
Done when: three tests pass and calls are counted.
Hint
catch GateFailed inside a loop and rebuild the prompt with the reason.
Medium 2: A parallel timeout (60 minutes)
Add a per-call timeout to parallel: a worker that takes too long goes into errors and the run returns on time. Test with one deliberately slow worker.
Done when: total time is near the timeout and the fast results are present.
Hint
future.result(timeout=...) raises TimeoutError. The thread keeps running, so think about cleanup.
Medium 3: Cheap router (60 minutes)
Replace the model classifier in route with a keyword function first, and use the model only when no keyword matches. Report calls saved over 20 tickets and any accuracy lost.
Done when: a table shows calls and accuracy for both versions.
Hint
put the keyword check in front and call the model only on "other".
Hard
Hard 1: A cost-aware orchestrator (120 minutes)
Give each subtask a cost estimate and a budget. The orchestrator drops the lowest-priority subtasks when the plan would exceed the budget, and says what it dropped. Reuse the ledger idea from Level 2, module 6.
Done when: a test shows a dropped subtask and the final answer mentions it.
Hard 2: A model evaluator (120 minutes)
Replace the code evaluate with a model call that returns PASS or feedback. With a real or imitation model, compare rounds and calls against the code evaluator. Add a guard that treats malformed evaluator output as a failure. (With a real model I did not run this, so your numbers are the ground truth.)
Done when: you have a table of rounds, calls and pass rates and a note on evaluator reliability.
Hard 3: The whole triage flow (120 minutes)
Build the Level 3 project skeleton: route the ticket, run the right handler to draft a reply, check the draft with parallel checks, and refine if a check fails. Report the calls per ticket on 20 sample tickets.
Done when: a test covers each stage, and you can state the average and worst case call count.
Hint
reuse route, parallel and refine from patterns.py. The glue is only a few lines.