Part 4 of 5 · 4 min read
Exercises
Easy
Easy 1: Run and read (20 minutes)
In code/mini-tools, run pytest and python demo.py. From section 1 of the demo, give the accuracy of the "good" descriptions on the 24 requests and on the 12 held-out requests, and explain the gap in two sentences.
Done when: you report both numbers and mention that the descriptions were written looking at the first set.
Hint
the first set was in view when I wrote the descriptions; the second was not.
Easy 2: Critique four tools (20 minutes)
Rewrite each description so a model could choose it correctly: (a) process(data): "Processes data."; (b) search(q): "Searches."; (c) delete_file(path): "Deletes a file."; (d) send(msg): "Sends a message to the user."
Done when: each rewrite says what it does, when to use it, when not to, and mentions any side effect.
Hint
for (c) and (d), say what is irreversible and who the "user" is.
Easy 3: Write the error (20 minutes)
Write the error message you would return for: (a) the model asks for a date 13/25/2026; (b) a file that does not exist; (c) a rate-limited search API; (d) a refund larger than the order.
Done when: each message names the problem and tells the model what to try next.
Hint
copy the pattern "wrong value for X: expected Y (example Z), got W. Try ...".
Medium
Medium 1: Add a tool and a held-out set (60 minutes)
Add reorder_product(sku, quantity) to the toolbox with a description, and add 4 requests for it plus 4 for each existing tool that might get confused with it. Write the requests before you finish the description. Report accuracy before and after adding the tool.
Done when: you have two accuracy numbers and a note on which existing request got worse.
Hint
write requests first, then freeze them, then edit descriptions.
Medium 2: A confirm flow for cancel (60 minutes)
Give cancel_order the same preview-then-confirm behaviour as refunds, plus a rule that shipped orders cannot be cancelled with an error that points to refund_order. Add tests for preview, confirm, shipped order and unknown order.
Done when: four tests pass and the "shipped" error names the alternative tool.
Hint
use the same confirm pattern as the refund; the preview must not change any state.
Medium 3: Retrieve a few tools (90 minutes)
Build 40 tools (12 real plus 28 plausible ones from other domains). Add a step that scores all tool descriptions against the request and gives the model only the top 5. Measure how often the correct tool is in the top 5, and the selection accuracy with and without the filter.
Done when: you report top-5 recall and a before and after accuracy, and show one request where the filter removed the right tool.
Hint
embed or word-match descriptions; keep the top 5; log misses.
Hard
Hard 1: Real tool choice (150 minutes)
Use a real model with native tool calling. Build the four description variants of section 2 for a toolbox of your own (at least 10 tools), write 40 requests plus 20 held-out, and measure tool choice accuracy on each. (I did not run this, so your numbers are the ground truth.)
Done when: you have a table like section 2 from a real model, with the held-out gap and the three most common wrong picks explained.
Hint
set temperature to zero and run each request three times to see the variation.
Hard 2: Error messages with a real model (120 minutes)
Take 30 realistic bad calls from a real model (collect them from running a real agent, not invented). Return the three error styles of section 3 and count how many calls are fixed within two retries, and how many tokens each style costs.
Done when: you have a recovery table by style, a token count per style, and a recommendation.
Hint
count a call as fixed only if the second or third call actually succeeds, not just validates.
Hard 3: A safe email tool (150 minutes)
Design send_email with schema validation, an allow-list of domains, a preview step, an idempotency key, a rate limit and an audit log. Build a fault injector that loses replies, repeats calls and sends malformed arguments, and prove with tests that no fault can send an email twice or to a disallowed domain.
Done when: a test suite with at least eight fault cases passes and the audit log reconstructs every attempt.
Hint
make the key part of the schema, not an optional extra.