Skip to content
Call wiki

What can an AI do driving your browser, and when is that the right tool?

Last updated 2026-08-18 · Overview — no new claims; each statement links to the entry that carries the primary sources.

“The AI drives your browser” means exactly that: it takes a screenshot of the page, decides where to click or type, and repeats — Claude for Chrome, the Codex extension, and their embedded browsers all work this way (Browser agents).

Anything clickable

Section titled “Anything clickable”

A browser agent can do roughly whatever you can do on a website — read, click, fill forms, manage tabs (Browser agents). What I tell people on calls: if you can do it on a website, it can do it; the real limiter is whether you can pre-specify what you want from the site.

Also the slowest route — reach for it last

Section titled “Also the slowest route — reach for it last”

Screenshot-and-click is slow and token-hungry next to a connector or API (Browser agents). Doing email through the browser when the Gmail connector exists is, as I put it on a call, cursed — screenshots versus talking in code to Gmail. And many GUI-looking apps sit on a protocol or API an agent can script directly; ask “can this be done from the terminal instead of the app?” before accepting screen control (GUI to protocol). My order: connector first, code second, browser last.

Where it earns its place

Section titled “Where it earns its place”

Sites nothing else reaches. Plain web fetches fail on login walls, bot defenses, and JavaScript-rendered pages — and can silently fabricate from search snippets when they fail (Web fetch limits). The browser covers those, plus actions rather than reads: flight check-in, cookie banners, Apple Notes through iCloud’s web app when no connector exists (Browser agents).

It comes in two forms: your real Chrome, which acts with your logins, and a credential-less embedded browser that risks none of them (Browser agents). Logged-in state means your Chrome; anonymous research fits the embedded one.

How it’s contained

Section titled “How it’s contained”

Site permissions are the lever: the agent asks per site, re-prompts for protected actions even on always-allowed sites, and blocks categories like banking outright; for an individual, blocking the few must-not-touch sites is lighter than keeping an allowlist (Browser agents) — on one call that meant blocking a patient-data backend and leaving the rest open. Blocking a site there doesn’t touch connectors, which reach services server-side. The usual approval ladder still applies (Permission modes), and browser use often ships beside computer use and full access, each widening the blast radius (Agent isolation).

What to watch

Section titled “What to watch”

Prompt injection is heavily demonstrated, but the record of real personal-agent harm is still empty — calm that’s dated, not permanent (Agent safeguards). The failure I’ve actually seen bite is under-specification: a browser agent handed a vague, aggressive goal once emailed a friend’s entire customer list. So supervise the first run of anything new, keep hard rules in standing instructions (“never send an email unless I explicitly ask”), and lean on safeguards outside the model — approval prompts, scoped site permissions — not its promises (Agent safeguards).

Turn it into the tester

Section titled “Turn it into the tester”

Call advice, no entry behind it: when the agent builds something web-facing, make the browser its QA tool — have it open the page, take screenshots, and test the main flows itself instead of you clicking through every fix.

Go deeper

Section titled “Go deeper”