Part 1 of 4 · 1 min read
Overview
Many useful tasks sit behind interfaces with no API. A computer-use agent can reach them, but each step costs a model call and an observation, and each step can fail; the representation you choose sets cost, precision and what the agent can reach. Knowing the arithmetic lets you decide when a screenshot is worth its tokens.
About the example, and what is real
Real: the screen generator, a normalised cross-correlation grounder (with and without sub-pixel phase search), the click-precision formula with a simulation check, the token and compounding arithmetic, and 20 tests in code/mini-multimodal. Assumed: glyph patterns as "text", 192 x 192 screens with six buttons, pixel noise levels, the 28-pixel tile and 4-characters-per-token constants, the 150-element page, the $3 per million price, the 0.98 DOM success and the 80 percent failure-detection rate. Not real: no multimodal language model, no real browser or page, no real DOM; a language model's visual errors are not Gaussian pixel noise, and real text is harder to read than my glyphs.