Part 7 of 9 · 4 min read
The interview angle
Covers: The interview angle; A checklist for "ready to let an agent browse for a person".
The interview angle
How does an agent read a web page: DOM or screenshot?
Both are used, and they fail differently. The page source or accessibility tree is cheap, searchable, gives names and addresses of controls, and shows hidden text a person cannot see, which is the injection surface. A screenshot is what a person sees, handles pages the text cannot describe (painted text, canvases, icons), and costs a fixed number of tokens per look, but it needs coordinates (so it needs scaling and zoom handling), cannot see below the fold or under a pop-up, and misses small targets. A reliable system usually mixes them. Say what you know about what your framework strips from the source.
How would you secure a browser agent that can reach a user's accounts?
Do not rely on the model to resist. Run the browser in an isolated environment with minimal privileges; keep credentials out of the model's context and fill them in the browser by host; limit the hosts the agent may reach by exact match; fix the writes a task may make before reading any page, and give read-only tasks none; ask the person to confirm irreversible actions with details built from the request, and ask rarely; give the agent only what the task needs; log everything. Then add the model-side defences (training, classifiers, red-teaming) as a layer, not as the layer. Say what stays unsolved: an instruction that asks for something inside the task, and a person who stops reading prompts.
Why can't you just filter out the malicious text?
Because the attacker writes in the same language as the data, and a filter is a guess. A pattern list catches the wording you thought of, a classifier catches more and still misses some, and hidden text can sit where your filter does not look (a label, an image, painted pixels). Filters help; put the control where the action happens (host, contract, confirmation, credential), where you do not need to understand the sentence.
How would you evaluate a browser agent?
On the state of the world, not on what the agent says. A task has a success test (the order exists, the mail went to the right person) and a harm test (nothing happened that was not asked for, no private value reached the wrong host). Run the same tasks under page features that break agents (banners, late loading, icon-only buttons, hidden look-alikes), under injected instructions in several places, and report each separately; success and harm are different columns.
A checklist for "ready to let an agent browse for a person"
- [ ] The agent runs in a separate environment with only what it needs, not on the person's own machine and sessions.
- [ ] Credentials stay outside the model: filled by the browser, per host, never typed from the model's context.
- [ ] Requests go only to named hosts, by exact match, for reads and writes.
- [ ] The writes a task may make are decided before any page is read; read-only tasks have none.
- [ ] Irreversible actions (pay, send, transfer, delete, accept terms) wait for the person, with details built from the request.
- [ ] The agent reads only what a person could read, and acts only on what a person could click, unless you have a reason.
- [ ] Screenshot coordinates are scaled back, tested at every size you use, and small targets are handled by zoom or by the page.
- [ ] There is a step cap and a repeat stop.
- [ ] CAPTCHAs and two-factor prompts stop the run and ask.
- [ ] You know what each of the layers does not stop, and which attacks are inside the task.
- [ ] You have planted defects to see which checks would notice, and you know your suite's escapes.
- [ ] You measured the model's resistance on your pages, and know it is a probability.
Check yourself
A page shows a "Pay now" button while hidden text tells the agent to click it. The task was to read prices. What stops the click? (The task contract: payment is not in scope, so the harness refuses it and asks the person. The model should not be the one deciding.)
For the curious Three views of one page
The page source shows hidden text, the accessibility tree shows what assistive tools see, and a screenshot shows what a person sees. Each reads different things, so a page can say one thing to a person and another to an agent. Screenshot coordinates must be scaled to the real screen, or every click misses.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Judge the action, not the claim | Look at what the click would do, then allow or ask. |
| A host allow list | The agent can only visit sites the task needs. |
| A password vault | The model never sees the password. |
Draft: confirm which browser tool your agents use and how people approve actions.