A browser agent is an agent whose tools are the controls of a web browser: load a page, read it, click, type, scroll, go back. A computer-use agent does the same for a whole desktop, so it can also work in native applications. An observation is what the agent is given about the current state: the page's source, its accessibility tree, a screenshot, or a mix. An action is one input to the interface: a click at a position or on a named element, some typed text, a key, a scroll, a wait.
The loop is short: observe, decide, act, observe again. Anthropic's documentation for its computer-use tool (I read it on 24 October 2026, through a summary my fetch tool produced, so check the page for exact figures) describes it as: the model returns tool calls, your program runs them in an environment you control and sends back the results, and the loop ends when the model asks for no more. It tells you to set a maximum number of iterations to prevent runaway loops and cost. The same page lists about seventeen actions (screenshot, zoom, click variants, drag, scroll, type, key, hold a key, wait), and says every coordinate is in the pixel space of the screenshot you returned.
Two design choices set everything else in this module.
What does the agent see? The source and the accessibility tree are text and can be searched, but they are not what a person sees. A screenshot is what a person sees, but it is pixels the model must read, and it carries no names or addresses of controls.
What does the agent point at? An element (by an identifier from the tree) or a position (by coordinates in the image).
Section 2 makes the first choice concrete; section 5 the second.
Three ways of looking at one page
The same instruction hidden eleven ways in a product listing. Red: the agent reads it. Green: that view never sees it.
The example turns a page into a list of items (words the agent can read, each with a handle to click) in three ways.
View
What it contains
Handle
What it cannot do
Page source
every element, every attribute, hidden text, labels, comments, test identifiers
element id
cannot read words painted as pixels; cannot tell what a person would see
Accessibility tree
roles and names of what a screen reader would announce
element id
cannot see unnamed icons, painted words, or text marked aria-hidden; does not know which element is covered
Screenshot
the words drawn in the window, with positions
pixel coordinates
cannot see below the first screen until it scrolls, anything under a pop-up, or any name that is not drawn
My source view is the raw source, which is the worst case. Many frameworks hand the model a filtered list (for example, only visible, clickable elements, numbered). To my knowledge that is how some of them work, but I did not check any in this session, so find out what yours strips. A fourth view exists too: the plain text a script gets from a page (innerText). I measured it, but my agents do not use it (it cannot click).
Is my page model right? A test against Chromium
The rules for what each view sees are easy to get wrong, and an agent's security rests on them. So real_browser.py renders my pages in headless Chromium through Playwright and measures, for twelve cases, whether the words appear in the page's text, in the accessibility snapshot, and in the pixels (a screenshot compared with one of the same page without the words; text under six pixels tall counts as unreadable).
Case
In innerText
In accessibility tree
Readable on screen
plain text
yes
yes
yes
display:none
no
no
no
visibility:hidden
no
no
no
white on a white background
yes
yes
no (the pixels are identical)
positioned off screen
yes
yes
no
1-pixel font
yes
yes
no (2 pixels changed, not readable)
opacity:0
yes
yes
no
aria-hidden="true"
yes
no
yes
aria-label on a link that has words
no
yes
no
alt text on an image
no
yes
no
words painted on a canvas
no
no
yes
HTML comment
no
no
no
All twelve agree with my model (check C31, and four tests, skip themselves if no Chromium is installed). Three rows are worth remembering. White-on-white text is invisible to a person and a screenshot, and fully visible to every text-based view.aria-hidden text is the reverse: drawn on screen, absent from the tree, so an agent on the tree cannot see a price a person can. And an aria-label is read by the tree (it becomes the name) and not by innerText or the screen, which makes it a good place to hide an instruction from a person who uses a screenshot or reads the page.
Tiny text is a measurement, not a rule
A 1-pixel font changed two rows of pixels in Chromium, so "invisible to the screenshot" is only true because a reader cannot make out text two pixels tall. A model reading a high-resolution screenshot, or one that zooms into a region, might. The documentation I read says its zoom action inspects a region at full resolution. Treat "not drawn legibly" as a property of the resolution you chose.
The harness loop: judge the action by what it would do
One action through the harness. The agent proposes; the harness works out what the action really is; layers decide; the person confirms the irreversible ones.
Every click goes through two steps. plan_click works out which element receives the click and which request it would send: for a position, it does the hit test a browser does (top layer first, then the smallest box); for an element, it reads the element's own action. That description, an Effect, is what the layers look at. Only then does execute send the request.
Why not trust the agent's description ("completing your order")? Because the description comes from the thing you are defending against. A page that can steer the model can steer its account of what it is doing. The request, the host, the parameters and the element's own words are facts about the browser's state that the page cannot rewrite. The notes return to this in section 8, where one planted defect shows the confirmation prompt built from the agent's words instead.