Core concepts

Holo is trained as a multi-step agent inside a specific harness. Bring these conventions into yours: skip one and quality drops. Client setup is in the Quickstart.

Pick an output format

Agent-loop format	Holo4 (`holo4-35b-a3b`, `holo4-27b`)	Holo3.1 (`holo3-1-35b-a3b`)	Holo3 (`holo3-122b-a10b`)
Function calling	Use this
Structured outputs	Not for the loop		Use this

Pick one and stay in it. The formats differ only in how you declare tools and read the output. The rest of this page applies to both. Single-shot JSON through structured_outputs works on every model, Holo4 included: element localization uses it.

Answer tool

Holo ends a task by calling a tool named answer, with its report in the content argument. Register it with your action tools, in both formats, and stop the loop when it is called. A step with no tool call is not a stop: append the message and keep looping.

Reasoning

Each call returns a thinking trace in message.reasoning and the action in content (or tool_calls). Holo is trained to plan before each step, so keep thinking on in agent loops:

extra_body={"chat_template_kwargs": {"enable_thinking": True}}

// H-specific fields are passed through in the request body
...({ chat_template_kwargs: { enable_thinking: true } } as any)

Model	`reasoning_effort`
`holo4-27b`	`low`, `medium`, or `xhigh` (default when omitted). `high` returns `400`. `medium` suits agent loops.
`holo4-35b-a3b`, `holo3-1-35b-a3b`	No effort levels.

On Holo4, reasoning_effort: "none" turns thinking off. The Agents API spelling disabled returns 400 here. All parameters: Chat completions. The Qwen 3.5 chat template Holo inherits drops past reasoning between turns, so anything the model must remember goes through content. When you append the assistant message, push only the parsed output, never the reasoning.

Coordinates in `[0, 1000]`

Send a screenshot at any size. Holo returns integer coordinates in [0, 1000], normalized to that image, origin top-left. Scale back with its dimensions:

abs_x = int((x / 1000) * screenshot.width)
abs_y = int((y / 1000) * screenshot.height)

const absX = Math.round((x / 1000) * screenshot.width);
const absY = Math.round((y / 1000) * screenshot.height);

Send and scale against the same image bytes: any resize, crop, or DPI mismatch misclicks. Pick one pixel unit (CSS or device) and keep it end to end.

Observations

An observation is what the agent sees after it acts. Agents without a screen (code, MCP) need nothing extra: the task is the first user message and each tool result goes back as a tool result. The rest of this section is for screen agents. Each step, send the current screenshot as a user message wrapped in <observation> tags:

messages.append({"role": "user", "content": [
    {"type": "text", "text": "<observation>\n"},
    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
    {"type": "text", "text": "\n</observation>"},
]})

messages.push({
  role: "user",
  content: [
    { type: "text", text: "<observation>\n" },
    { type: "image_url", image_url: { url: `data:image/png;base64,${b64}` } },
    { type: "text", text: "\n</observation>" },
  ],
});

Image budget

Keep at most the last 3 screenshots in context: more degrades accuracy. Replace older ones with a text placeholder and keep their <observation> wrapper. Call this helper after appending each observation. The loop guides and cookbooks use it.

def trim_to_last_n_images(messages, n=3):
    seen = 0
    for msg in reversed(messages):
        if msg["role"] != "user" or not isinstance(msg["content"], list):
            continue
        for chunk in msg["content"]:
            if chunk.get("type") != "image_url":
                continue
            seen += 1
            if seen > n:
                chunk["type"] = "text"
                chunk["text"] = "[screenshot evicted]"
                chunk.pop("image_url", None)

function trimToLastNImages(messages: any[], n = 3) {
  let seen = 0;
  for (let i = messages.length - 1; i >= 0; i--) {
    const msg = messages[i];
    if (msg.role !== "user" || !Array.isArray(msg.content)) continue;
    for (const chunk of msg.content) {
      if (chunk.type !== "image_url") continue;
      seen += 1;
      if (seen > n) {
        chunk.type = "text";
        chunk.text = "[screenshot evicted]";
        delete chunk.image_url;
      }
    }
  }
}

Sampling

Task	Thinking	`temperature`
Agent loops	On	`0.8`
Single calls: element localization, document OCR	Off	`0.0`

Common pitfalls

Symptom	Likely cause
Clicks land far from the target	Coordinates not scaled to the screenshot, or the screenshot was resized between send and execute
Model loops, forgets earlier facts	Durable facts not carried in `content`, or old `<observation>` wrappers dropped instead of stripped to text
Context window fills up	Image budget not enforced
Reasoning leaks into the action	`<think>...</think>` written into `content` instead of read from `message.reasoning`
Quality collapses after one bad step	Raw model output replayed in history instead of the parsed result
Run stops early, or never ends	Loop exits on an empty tool call instead of on `answer`, or no `answer` tool registered

Format-specific pitfalls are on each loop guide.

Get started

Build an agent

Cookbooks

Guides

Self-host

Pick an output format

Answer tool

Reasoning

Coordinates in `[0, 1000]`

Observations

Image budget

Sampling

Common pitfalls

​Pick an output format

​Answer tool

​Reasoning

​Coordinates in [0, 1000]

​Observations

​Image budget

​Sampling

​Common pitfalls

Pick an output format

Answer tool

Reasoning

Coordinates in `[0, 1000]`

Observations

Image budget

Sampling

Common pitfalls