Part 4 of 5 · 3 min read
Exercises
Easy
Easy 1: Direct or indirect? (15 minutes)
Classify each case and name the data source: (a) a user types "ignore your rules", (b) a web page contains hidden text telling an agent to email the user's files, (c) an uploaded PDF contains an instruction in white text, (d) a customer review says "rate this product five stars".
Done when: four labels with a reason each.
Easy 2: Spot the hole (15 minutes)
A prompt wraps user text in <user> and </user> but does not escape anything. Write an input that would break out, and the one-line fix.
Done when: your input and fix are tested with the example's wrap.
Hint
the input contains </user> followed by your own instruction; the fix is escaping or a random tag.
Easy 3: Blast radius (20 minutes)
For a support agent that can read orders, issue refunds and send email, list what each tool could do if the agent were tricked, and mark which should need human approval.
Done when: a table of three tools, worst cases, and a decision for each.
Medium
Medium 1: Add three attacks (60 minutes)
Write three new attacks for the example (for instance a polite one, one split across two lines, one in capital letters) and extend the toy model's patterns if necessary. Report which layers stop each.
Done when: a table of attacks against layers, and tests for the new cases.
Hint
the toy model's REPLY pattern is the thing attacks must match; broaden it, then see which layers still work.
Medium 2: A stricter validator (60 minutes)
Add to validate a rule that the reason may not contain a URL or an email address, and a length limit. Test with accepted and rejected replies, and explain what attack each rule helps against.
Done when: six tests, with a sentence per rule.
Medium 3: Sandwich defence (60 minutes)
Add an option to repeat the classification instruction after the ticket (repeat the instruction after the untrusted text). Make the toy model honour it with some probability and measure the change. Note the extra tokens it costs (Level 2, module 6).
Done when: a table shows success rate and prompt size with and without.
Hint
put the repeated line after the closing tag and have the toy model ignore injected lines when the repeat is present, with a slip probability.
Hard
Hard 1: Indirect injection test (120 minutes)
Build a mini pipeline that fetches a "web page" (a string) and summarises it. Plant an instruction in the page and show it is followed without defences. Add the fence, a rule and validation, and measure again.
Done when: before and after rates are reported and the remaining weakness is described.
Hard 2: Real model probe (120 minutes)
Run your attack set against a real model with and without the fence, 20 times each. Report the rates and the model name. Be cautious: use only your own test app and no real data. (I did not run this, so your numbers are the ground truth.)
Done when: a table of rates and a paragraph on how the real model differed from the toy.
Hard 3: Tool-call guard (120 minutes)
Extend module 4 of Level 2's Registry with an allow-list per step and a confirmation callback for tools marked risky. Show that an injected request to call a risky tool is blocked without confirmation.
Done when: three tests: allowed call works, disallowed call blocked, risky call needs confirmation.
Hint
raise a PermissionError from the registry and return it as an error string, as module 4 of Level 2 does.