Computer use: when the AI clicks

Until now an assistant supplied text and you clicked. With computer use it clicks itself: it sees your screen, moves the pointer, types into fields. That sounds like the solution to everything – and in practice is a specialist tool.

The loop behind it

The model receives a screenshot and your brief. It decides on the next step – “click at position X”, “type this text” – the program carries it out, and a new screenshot shows the result. Then the loop starts again.

The properties you need to know follow from that:

• Slow. Every click is a separate model call. What you do in ten seconds takes minutes.
• Error-prone on small things. Tiny type, similar buttons and unexpected dialogs are the typical stumbling blocks.
• Universal. It works with any program that has a window – including the old accounting package with no interface.

The order of tools

Both major providers describe screen control as the route for cases where nothing better fits. The order:

1. Direct integration – connector or interface. Fast and accurate.
2. Command line – for anything that can be expressed as a command.
3. Browser integration – for websites; it reads the page structure rather than pixels. More on that in the next station.
4. Computer use – for programs with no other door.

Anthropic states explicitly for Claude that tasks via screen control run more slowly than through connectors and that Claude only falls back on it when none of the other options fits.

Where ChatGPT and Claude stand

OpenAI does not list the supported regions individually on its help page – whether the feature appears in your account is something you see in the settings. With Claude, Team and Enterprise plans do not have access yet. ChatGPT’s former “agent mode” with its own virtual computer has been discontinued; OpenAI points to ChatGPT Work instead. Computer use at OpenAI · Computer use in Cowork · Claude Code on desktop · ChatGPT agent (discontinued)

What it deliberately does not do

According to OpenAI’s documentation, computer use does not automate terminal programs or ChatGPT itself, does not sign in as an administrator and does not approve the system’s security or privacy prompts. Those are the right limits: a tool that could grant itself more rights would no longer be a tool.

What it is worth using for

Good: a clearly outlined task in one app where a mistake shows and can be fixed – renaming files, going through settings, typing data out of a legacy program, clicking through and testing your own software.

Bad: anything involving money, accounts or the irreversible. And anything that could be done in one go through a connector.

Four rules for the first attempt

1. One app, one flow. The narrower the brief, the fewer forks.
2. Close what is not needed – above all inbox, password manager and banking.
3. Stay present as soon as accounts, payments or security settings are involved. OpenAI recommends exactly that.
4. “Always allow” only for programs you trust – and tidy the list regularly.