Skip to main content
With local desktop control, the agent works on the machine in front of you. It drives your real mouse, keyboard, and screen, with the same agent and session lifecycle you already use. Run a session that uses a local desktop from the hai-agents Python SDK (the only SDK with local control today) and it connects your machine to H for you. Works on macOS, Windows, and Linux. Use it when the task lives outside the browser: native apps, the file system, or multi-window flows on a machine you control.
The agent controls your whole desktop: it can move the mouse, type, read anything on screen, run shell commands, and read or write files. Prefer a dedicated machine over your primary one and run it only for tasks you would do yourself. Stop a run at any time by cancelling the session.

​
How it works

A local desktop is an environment with its kind set to desktop and host set to user_device. When a session starts with one, the SDK opens a connection inside your Python process. The connection receives the agent’s actions and carries them out on the real desktop, and it is what ties this particular session to this particular machine. Everything else is unchanged: observe and steer the run and read its answer just as you would a remote one. A Python process serves one local desktop session at a time. Starting a new session that uses the local desktop while an earlier one is still running hands the desktop to the new session and cancels the earlier one.
1

Install the desktop driver

The desktop driver is an optional extra, provided by the hai-drivers package:
Install
pip install "hai-agents[desktop]"
2

Run a session

In Python, define the environment inline on the agent and run a session. There is nothing else to set up; the SDK connects the desktop before the session starts. From the CLI, serve the desktop with hai local desktop and point the agent at the session_id it prints.
from hai_agents import Client

client = Client()  # reads HAI_API_KEY from the environment

result = client.run_session(
    agent={
        "name": "local-desktop",
        "description": "Drives the desktop on my own machine.",
        "environments": [
            {
                "id": "my-laptop",
                "kind": "desktop",
                "host": "user_device",
            }
        ],
    },
    messages="Open Notes and write a short summary of today's standup",
)
print(result.status, result.answer)

pip install "hai-agents[cli,desktop]"

# Serve the desktop and leave it running; it prints the session_id it serves.
hai local desktop

# From another terminal, route a registered agent's desktop environment here.
hai run "Open Notes and write a short summary of today's standup" \
  --agent my-desktop-agent \
  -o 'agent.environments[kind=desktop].session_id=<printed id>'
The session behaves like any other: observe and steer it, read changes, or watch it in Agent View. The Python connection closes when your process exits.Auto-connect covers agents defined inline in run_session, start_session, or create_session. A registered agent referenced by name, or a session started from the web app or another machine, expects its user_device environment to carry the session_id of a machine served with hai local desktop, as in the CLI tab. Set HAI_AUTO_BRIDGE=0 to opt out of auto-connect entirely.
3

Grant OS permissions when prompted

The agent controls the real mouse and keyboard and reads the screen, so your operating system has to trust the program running it (your terminal, or the app that launches Python). On macOS, your first session triggers two permission prompts. Grant both in System Settings → Privacy & Security, then restart the program and run the session again:
  • Accessibility, to move the mouse and type.
  • Screen Recording, to read the screen.
On Windows and Linux there is nothing to grant; run the program in a normal desktop session so it can reach the display.

​
Next steps

Local browser

Drive Chrome on your machine the same way.

Observe & steer

Watch a local run, redirect it mid-task, and read the answer.

SDKs

Install the clients and authenticate.

Environments

How environments attach to an agent and what each kind contributes.