Part 6 of 7 · 3 min read
Exercises
Each level has three exercises. Do the easy ones first.
Easy
Easy 1: Read the schema (15 minutes)
Print default_registry().schemas() and find, for each tool, its name, description, argument names and which are required.
Done when: you can describe both tools from the schema alone.
Easy 2: Break it on purpose (20 minutes)
Call REG.execute directly with five bad requests: unknown tool, invalid JSON, a missing argument, an empty city, and a city with no data. Print the five error messages.
Done when: each message tells the model what to fix.
Easy 3: Count the messages (15 minutes)
For a run that asks two tools at once and then answers, list the roles of all messages in order. Then do it for a run that makes three sequential single calls.
Done when: you have two lists, and the second has 8 messages.
Hint
three sequential calls make user, (assistant, tool) x 3, assistant = 8 messages.
Medium
Medium 1: Add a tool (60 minutes)
Add convert_units(value, from_unit, to_unit) for a few lengths and temperatures, with a Pydantic argument model that restricts units to an enum. Test valid conversions, an unsupported pair and a bad unit string, checking the error text.
Done when: five tests pass, including one where the error lists the supported units.
Hint
restrict units with an Enum, so Pydantic lists the allowed values in its error.
Medium 2: A call budget (60 minutes)
Extend run_with_tools with a max_tool_calls limit across the whole run (in addition to max_steps) and a stopped reason tool_budget. Test that a model asking for four calls per step stops once the total passes the limit, and that completed calls are not lost.
Done when: both limits are tested separately.
Medium 3: Detect a repeat (60 minutes)
Detect when the model asks for the same tool with the same arguments three times in a row and stop with stopped = "repeat", returning a message that explains why. Test with the looping model and with a model that legitimately calls the same tool with different arguments.
Done when: the repeat is caught and the legitimate case is not.
Hint
keep a list of the last three (name, arguments) pairs.
Hard
Hard 1: Parallel execution (90 minutes)
Run the calls of one step concurrently with asyncio (the fake tools can await asyncio.sleep). Keep the tool messages in the order of the call ids. Test that two 0.1-second tools finish in about 0.1 seconds, that one failure does not cancel the other, and that results are matched to the right ids.
Done when: three tests pass, including the timing.
Hint
asyncio.gather(*calls, return_exceptions=True) keeps one failure from cancelling the rest.
Hard 2: A permission layer (120 minutes)
Add a per-tool policy: allow, deny or ask. For ask, the loop pauses and calls a function you pass in (a fake human). A denied or refused call returns an error result saying so, without running the function. Test all three policies, and that the model sees a sensible message when denied.
Done when: tests cover allow, deny, ask-yes and ask-no, and no denied function body ever executes.
Hint
check the policy inside execute, before tool.fn is touched.
Hard 3: Evaluate tool choice (120 minutes)
Write 20 user requests with the correct tool and arguments for each (including 5 that need no tool). Run a model function (an imitation or, if you have access, a real one) over them and report tool-choice accuracy, argument accuracy and the rate of unnecessary calls. Include a confusion table of which tool was chosen instead of which.
Done when: you have three rates and a confusion table, and you propose one change to a description that you could test.