Architecture

Think of HoloDesktop CLI as a thin client around a local computer-use agent runtime. There are two main pieces:

• the open-source Python client, distributed as holo-desktop-cli;
• the computer-use agent runtime executable, hai-agent-runtime.

The client is the part you run from your terminal, MCP host, ACP host, A2A client, or Python code. The runtime observes the screen, plans actions, clicks and types, and streams events back.

​

The same runtime behind every surface

The CLI, MCP server, ACP server, A2A server, and Python client all route work to the same local runtime. The surface changes how the agent is invoked. It does not create a separate kind of computer-use agent. A skill is not a separate surface: it gives an MCP host reusable instructions on when to hand off to HoloDesktop CLI. For commands and flags, use CLI reference. For host setup, use Agent hosts.

​

Runtime lifecycle

The Python client starts or attaches to hai-agent-runtime on loopback: if a healthy runtime is already listening on the target port it is reused, otherwise the client starts one. Both run on your machine. Cache paths, logs, token files, and run directories are listed in Paths and files.

​

Inference path

The runtime sends model inputs either to H’s Models API (hosted mode) or to an OpenAI-compatible endpoint you provide (local mode). Runtime, desktop control, and diagnostics stay on your machine in both modes. See Hosted or local models for setup and Security and privacy for what each mode sends.

​

Desktop control

The agent operates desktop state, not source code or APIs. It observes what is visible, plans the next action, and uses desktop tools to click, type, scroll, and switch apps. Foreground state matters. A CLI task can move focus and use the active desktop while it runs. Host integrations may make it feel like a tool call, but the underlying action is still desktop operation. The agent observes and acts; it does not verify. Check results with code, not another model call. The expense-report example shows the pattern.

​

User context

The client snapshots user context at run start and sends it to the runtime with the task. From ~/.holo/ it picks up:

• standing instructions (agents.md);
• memories and rules;
• installed skills.

These files shape behavior, so keep them specific and review them first when a run behaves unexpectedly. Exact paths are in Paths and files.

​

Run artifacts

The runtime emits events as it works. Clients use those events to print progress, stream updates, and debug failures. The same stream is also persisted locally as run artifacts. For event structure, use Debug a failed run. For storage and privacy, use Paths and files and Security and privacy.

​

Next steps