Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-multimodal (about two minutes).
Done when: you can explain each table in the results.
Easy 2: Token budget (30 minutes)
For a screen size you use, compute screenshot tokens at full, half and quarter size with the 28-pixel assumption, and the cost of a 15-step task keeping all observations versus the latest.
Done when: your numbers match screenshot_tokens and keep_all_tokens.
Easy 3: Click precision (30 minutes)
Compute P(hit) for a 20 px target with coordinate error 3, 6 and 12 px.
Done when: your numbers match hit_probability.
Medium
Medium 1: Marks instead of coordinates (3 hours)
Extend the example: detect candidate regions, number them, and simulate a chooser that picks the right mark with a given accuracy. Compare it with coordinate noise at equal difficulty.
Done when: you plot success against chooser accuracy and coordinate error.
Medium 2: Correlated failures (3 hours)
Replace independent step failures with a shared "hard page" factor and measure how retry benefit changes.
Done when: you report the gap from the p^n formula.
Medium 3: Crop and zoom (3 hours)
Add a second pass that crops around the best match at full resolution and re-checks it. Measure accuracy at 4x and 8x shrink.
Done when: you report the gain and the extra tokens.
Hard
Hard 1: Real agent comparison (8 hours)
Build a Playwright agent with a multimodal model and compare DOM-only, screenshot-only and hybrid on 30 tasks. (I did not run this, so your numbers are the ground truth.)
Done when: you report success, steps, tokens and dollars per task for each.
Hard 2: Injection test (8 hours)
Add a page with hidden instructions in text and in an image and measure how often your agent follows them, then add a defence. (I did not run this.)
Done when: you report rates before and after and which action checks stopped it.
Hard 3: Pipeline in PyTorch (8 hours)
Train a small convolutional network to locate labelled buttons on screens like mine and compare it with the template matcher across shrink factors. (I did not run this.)
Done when: you report accuracy by size and shrink for both.