Definitions

Definitions

The pipeline

Results

1. Resolution is a budget: shrinking a screenshot saves tokens and costs accuracy, and small targets pay first.
2. Coordinates are fragile; choosing among marked elements is not.
3. History dominates cost unless observations are dropped or summarised.
4. Per-step reliability compounds, so verification and retry beat marginal model gains.
5. The DOM is cheap and precise where it exists; a screenshot fallback covers the rest.

Design rules

• Prefer text (DOM or accessibility tree) when it exposes the targets; add screenshots for visual or missing content.
• Filter the DOM to interactive elements before sending it.
• Annotate screenshots with marks and ask for a mark, not coordinates.
• Size the screenshot to the smallest target you must hit; test at your real resolution.
• Keep only the latest observation unless history is needed; summarise the rest.
• Verify each action (expected change appeared) and retry once before escalating.
• Set a token and step budget per task and log cost per task.
• Sandbox, filter and confirm: isolated browser, redacted screenshots, least-privilege credentials, human approval for sensitive actions.
• Treat page content as data, not instructions.

Interview questions, with the follow-ups

Common mistakes

• Sending the whole DOM instead of the interactive elements.
• Asking a model for raw coordinates on small targets.
• Shrinking screenshots without testing the smallest target.
• Keeping every screenshot in the context.
• No verification after an action.
• Assuming the DOM covers every widget.
• Letting page text act as instructions.
• Reporting step accuracy as task accuracy.

What the example does not do

• No language model: the grounder is a template matcher with the exact template; real models read text, handle layout and also err in ways unlike pixel noise.
• Glyphs, not words: random binary patterns stand in for labels; real labels carry meaning, fonts and scale changes.
• One screen at a time: no scrolling, pop-ups, dynamic content, drag, or multi-step state changes.
• Independent steps in the compounding arithmetic; real failures correlate.
• Assumed constants: tile size, characters per token, price, DOM success and failure-detection rates.
• No security test, no real browser, no Playwright.
• No audio or video; the module covers screens and images only.
• Optional real experiment (I did not run this): pick a form-heavy site, run a Playwright agent three ways (DOM only, screenshot only, hybrid) on 30 tasks, and record success, steps, tokens and dollars.

For the curious History is the hidden cost

Keeping every earlier screenshot in the context makes input tokens grow quadratically: a ten-step task used 73,370 tokens against 13,340 when only the latest was kept (about $0.22 against $0.04 at an assumed $3 per million). Small targets break first: with a pixel matcher, 11 px buttons failed once the screenshot was shrunk 4x. The matcher, the token constants and the screens are synthetic, so none of this measures a real multimodal model.

What you must be able to say (one page)

• What "multimodal" means here. A language model predicts the next token from text tokens; a multimodal model predicts it from text and image tokens (or other modalities). CLIP, trained on 400 million (image, text) pairs found on the internet, is an embedding model, not a generator, and embedding models like it are backbones of generative ones such as Flamingo and LLaVA.
• A computer-use agent has three parts: an action planner (a model turning a task into interface actions), an action executor (Playwright for browsers, PyAutoGUI for desktops) and an interface representation (what the model sees). Planning is explicit (a full plan from the first state; good for static forms) or implicit (re-plan every step; good for dynamic pages such as flight booking).
• Three ways to show the interface (see the token table in the results): text (the DOM, or accessibility APIs; cheap and precise, misses visual context and needs a text interface), image (screenshots; works with any interface, costs many tokens, loses small text) and hybrid (the usual recommendation for production web agents). Desktop and mobile apps often have no runtime text view, so screenshots are the only option. Set-of-mark prompting and OmniParser annotate the screenshot with numbered boxes so the model picks a mark instead of regressing coordinates.
• Why a screenshot reader is fragile (my experiment). I generated synthetic screens of labelled buttons (glyph patterns stand in for text) and located a target with template matching, a pixel matcher given the exact template, not a language model. Clicks inside the correct button, at pixel noise 0.1 (80 trials per cell): 11 px buttons 1.00, 0.99, 0.23 and 0.01 at full size, 2x, 4x and 8x shrink; 18 px 1.00, 1.00, 0.88, 0.26; 25 px 1.00, 1.00, 1.00, 0.81; 32 px 1.00, 1.00, 1.00, 0.99. Small targets and low resolution break first.
• Details matter. A matcher that assumed pixel-aligned buttons (18 px) fell to 0.78 at 2x shrink and 0.33 at 4x, against 1.00 and 0.88 when it searched over sub-pixel phases. Pixel noise at 18 px and 4x shrink: 0.97 at noise 0.05, 0.88 at 0.1, 0.60 at 0.2, 0.47 at 0.4. Each cell has 80 trials, so differences under about 5 points are not reliable.
• Click precision. If the model outputs coordinates with Gaussian error sigma, P(hit) = erf(s / (2 sigma sqrt 2))^2 for a target of side s (checked by simulation, within 0.002). To hit an 11 px button 90 percent of the time you need sigma under about 3 px; a 32 px button tolerates about 8 px. Rounding to a grid of spacing f adds about f / sqrt 12 pixels (0.58 px at 2x, 1.15 at 4x, 2.31 at 8x). Marks, crops and element lists avoid this by turning coordinates into a choice.
• Cost is a design constraint (treat cost as infrastructure: capture screenshots only when necessary and use smaller models for routine steps). With assumed constants (one token per 28 x 28 pixel tile, 4 characters per token, a page of 150 elements of which 30 percent are interactive): a 1280 x 800 screenshot is 1,334 tokens, at half size 345, at quarter size 96; the full DOM is about 2,825 tokens, the interactive-only DOM about 848, the hybrid (filtered DOM plus half-size screenshot) about 1,193.
• History is the hidden cost. Keeping every earlier observation in the context makes input tokens grow quadratically: a ten-step task with full screenshots uses 73,370 tokens (about $0.22 at an assumed $3 per million) against 13,340 ($0.04) when only the latest is kept; half-size screenshots with only the latest use 3,450 ($0.01). Prompt caching (module 6) reduces the price of a repeated prefix but not the growth.
• Errors compound. Task success is p^n for n independent steps: 0.95 per step gives 0.77 after 5, 0.60 after 10 and 0.36 after 20. With a check that notices 80 percent of failures and retries once, the same figures become 0.94, 0.89 and 0.79; at 0.99 per step, 20 steps give 0.82 without retries and 0.96 with them. Verification after each action is worth more than a slightly better model.
• The DOM is not always there. If the DOM exposes only a share of targets (canvas widgets, custom components), DOM-only success falls with coverage: 0.49 at 50 percent coverage, 0.78 at 80, 0.93 at 95 (DOM success 0.98 assumed). A hybrid that falls back to a screenshot (0.875 in my experiment: 18 px buttons shrunk 4x) holds 0.927, 0.959, 0.975.
• Security is part of the design: prompt injection through page HTML or rendered content, credential exposure, supply-chain risk from extensions. Common measures are sandboxing, filtering screenshots and DOM captures for sensitive data, audit logs, short-lived least-privilege credentials, rate limits and user confirmation for sensitive actions. I did not test any of this; the structure of my example cannot show it.
• What to say in an interview: "I prefer the DOM or accessibility tree when it exists, add screenshots when it is incomplete or visual, annotate with marks so the model picks elements instead of coordinates, keep only the latest observation, verify every action's effect and retry once, budget tokens per task, and sandbox the agent and confirm sensitive actions."
• Honest limits: synthetic screens with glyph labels, a pixel matcher rather than a multimodal model, no real pages, no DOM noise model beyond a coverage number, assumed token and price constants, and no test of any real agent's success rate.