Part 8 of 9 · 5 min read
Exercises
Easy
Easy 1: Run it and read the matrices (30 minutes)
In code/mini-browser, run python3 -m pytest -q (about 20 seconds), then python3 demo.py. Open results.json and find the matrix vector_none. Run python3 real_browser.py if you have Chromium.
Done when: for five hiding places you list which of the three views read each, and for two of them you show the Chromium row that backs it up.
Hint
the reach matrix is E.reached_matrix(); the Chromium rows are real_browser.fidelity().
Easy 2: Move the cut-off (40 minutes)
In perceive.py change MAX_LABEL_WHEN_HYGIENE from 40 to 100. Re-run python3 -c "import experiments as E; print(E.harmed(('hygiene',)))" and the long-label variant.
Done when: you report which attack cells come back (the label and alt-text attacks are about 90 characters), whether the long-label task now succeeds, and explain why no cut-off is right: what the attacker does when you pick one.
Hint
the constant is used in source_view and a11y_view. Payloads are about 90 characters long, so the cut-off decides whether they pass.
Easy 3: Close a hole in the suite (40 minutes)
Three planted defects escape: hygiene_drops_aria_hidden, confirm_total_wrong, step_cap_off_by_one. Write a check C32 for one of them and add it to checks.py.
Done when: C32 passes on the correct harness, fails with the defect planted, and python3 evaluate.py now reports one fewer escape; add a test for it.
Hint
for confirm_total_wrong, build the prompt for a known cart and check the total in the text; for the step cap, count the steps that ran; for hygiene, look at what the source view returns for an aria_hidden price.
Medium
Medium 1: A twelfth hiding place (2 hours)
Find a way to hide text that is not in the list: text from a CSS ::before rule, a title attribute, an SVG <text> element, text under another element, or text in an iframe. Measure it with real_browser.py in the same three ways, add it to HIDING or as a new vector, and run the six attacks through it.
Done when: the Chromium measurement is in fidelity(), your model agrees with it (or you say where it does not), and you update the reach matrix and a test.
Hint
new hiding goes in dom.HIDING (three values: in the tree, drawn, CSS) and world.VECTORS; real_browser.to_html needs the CSS.
Medium 2: Verify the choice, not the type (3 hours)
The steer attack buys a USB-C cable that is not the cheapest. Write an independent verifier that fetches the search results itself (not through the agent), computes the cheapest USB-C cable, and lets the contract accept only that product. Make it a layer of its own.
Done when: steer drops from 6 harming cases to 0 with all layers on, the legitimate purchase still passes in every view, you add checks and a planted defect, and you say for which tasks no such verifier can be written (what the person asked for has no function that computes it).
Hint
the contract's check receives the parameters and the world; give it a function that fetches /search itself with the same Browser and reads the prices, so it does not depend on the agent.
Medium 3: A better detector, and an attack on it (3 hours)
Replace the pattern list with a scorer that looks for imperative sentences addressed to an assistant (verbs such as buy, send, forward, transfer, open, sign in, plus an addressee or a URL). Evaluate it on the 12 attack wordings and the 12 ordinary lines, then write three paraphrases of your own that the stand-in still obeys and your scorer misses.
Done when: you report precision and recall on both sets, show the three evasions, and say which layer would still have stopped each attack.
Hint
the detector is used in Guard.keep; the stand-in's reading rules are in agent.TRIGGER and RX, so your evasions must still match those.
Hard
Hard 1: A real browser behind the same interface (4 hours)
Implement the Browser interface over Playwright: pages served from local files (use to_html in real_browser.py), the accessibility view from aria_snapshot, the screenshot view from real screenshots, clicks by element and by coordinate with real scaling. Re-run Experiment 1. (I did not run this, so your numbers are the ground truth.)
Done when: you report the variant table for the real browser, say which cells differ from the simulation and why, and state whether the layers still hold unchanged.
Hint
the interface is goto, shot, plan_click, execute, type_text, scroll, wait, hit, covered; start with the source and the screenshot, add the tree last.
Hard 2: A real model through the three views (4 hours)
Replace the scripted plan with a real model, using each view, ten runs per page feature, and then repeat with the six attacks hidden in the places of section 6 on your own test pages. (I did not run this.)
Done when: you report success and harm per view with the model's name and the date, say how often the model refused or ignored an injected instruction with no layer on, and say which of the scripted conclusions held.
Hint
write the prompts and the success tests before you run, keep temperature fixed, and do not tell the model it is being tested.
Hard 3: Break the harness (5 hours)
Look for a way around one layer: a redirect chain, a form whose action points to another host, a link that opens a new window, a data: or javascript: address, a hostname with a trailing dot or mixed case, a click that lands on an element the plan did not look at. Write three attacks. (I did not run this.)
Done when: you report each result with evidence, fix the code for any that worked, add a check and a planted defect, and say which attack surprised you.
Hint
Guard._check sees the request the browser would send; think about what a request does not show (where a redirect goes next).