Core concepts

Holo is trained to act as a multi-step agent inside a specific harness. Five of its conventions have to come along for the model to behave well in yours; skip any one and quality suffers.

Output format

Function calling or structured outputs, depending on the model.

Answer tool

The only way the model signals it is done.

Reasoning

Read it, never replay it.

Coordinates

Normalized to [0, 1000], scale back yourself.

Observations

Screenshots in <observation> tags, last 3 kept.

Sampling

0.8 for loops, 0.0 for single calls.

Set up the OpenAI client first by following the Quickstart.

Pick an output format

	Holo3.1 (`holo3-1-35b-a3b`)	Holo3 (`holo3-122b-a10b`)
Function calling	Recommended	Not supported
Structured outputs		Only option

Pick one and stay in it. Only how you declare tools and read the output changes; everything on this page is shared.

Answer tool

Holo is trained to end a task by calling a tool named answer with its work report in the content argument. Register it alongside your action tools, in both formats, and stop the loop when it is called. A step that comes back without any tool call is not a stop signal: treat it as a no-op, append the message, and keep looping. Only answer ends the run.

Reasoning

Holo returns two streams on every call: a thinking trace in message.reasoning and the action in content. Reasoning is essential in agent mode (Holo was trained to plan before each step), so leave it on; reasoning_effort: "medium" is a sensible default.

extra_body={"chat_template_kwargs": {"enable_thinking": True}}

// H-specific fields are passed through in the request body
...({ chat_template_kwargs: { enable_thinking: true } } as any)

Past reasoning is dropped between turns by the Qwen 3.5 chat template Holo inherits, so anything the model needs to remember has to flow through content. When re-adding the assistant message to the conversation, push only the parsed output. Do not splice the reasoning back in.

Coordinates in `[0, 1000]`

Send a screenshot at any size. Holo returns coordinates as integers in [0, 1000], normalized to that image. Scale back to pixels using its dimensions:

abs_x = int((x / 1000) * screenshot.width)
abs_y = int((y / 1000) * screenshot.height)

const absX = Math.round((x / 1000) * screenshot.width);
const absY = Math.round((y / 1000) * screenshot.height);

Origin is top-left. Send and scale against the same image bytes. Any resize, crop, or DPI mismatch will misclick. Pick one pixel unit (CSS or device) and stay in it end to end.

Observations

Each step, the current screenshot goes in as a user message wrapped in <observation> tags:

messages.append({"role": "user", "content": [
    {"type": "text", "text": "<observation>\n"},
    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
    {"type": "text", "text": "\n</observation>"},
]})

messages.push({
  role: "user",
  content: [
    { type: "text", text: "<observation>\n" },
    { type: "image_url", image_url: { url: `data:image/png;base64,${b64}` } },
    { type: "text", text: "\n</observation>" },
  ],
});

Keep at most the last 3 screenshots in context. More degrades accuracy. Replace older screenshots with a short text placeholder and keep the <observation> wrapper:

def trim_to_last_n_images(messages, n=3):
    seen = 0
    for msg in reversed(messages):
        if msg["role"] != "user" or not isinstance(msg["content"], list):
            continue
        for chunk in msg["content"]:
            if chunk.get("type") != "image_url":
                continue
            seen += 1
            if seen > n:
                chunk["type"] = "text"
                chunk["text"] = "[screenshot evicted]"
                chunk.pop("image_url", None)

function trimToLastNImages(messages: any[], n = 3) {
  let seen = 0;
  for (let i = messages.length - 1; i >= 0; i--) {
    const msg = messages[i];
    if (msg.role !== "user" || !Array.isArray(msg.content)) continue;
    for (const chunk of msg.content) {
      if (chunk.type !== "image_url") continue;
      seen += 1;
      if (seen > n) {
        chunk.type = "text";
        chunk.text = "[screenshot evicted]";
        delete chunk.image_url;
      }
    }
  }
}

Both loop guides call this helper right after appending each observation.

Sampling

Agent loops run with temperature: 0.8. Single-call tasks such as element localization and document OCR run with thinking off and temperature: 0.0.

Common pitfalls

Symptom	Likely cause
Clicks land far from the target	Coordinates not scaled to screenshot dimensions, or the screenshot was resized between send and execute
Model loops, forgets earlier facts	Durable facts not carried forward in `content`, or older `<observation>` wrappers dropped instead of stripped to text
Context window fills up	Image budget not enforced
Reasoning leaks into the action	`<think>...</think>` written inline in `content` instead of read from `message.reasoning`
Quality collapses after one bad step	Raw model output replayed in history instead of the parsed result
Run stops early, or never ends	Loop exits on an empty tool call instead of on `answer`, or no `answer` tool registered

Format-specific pitfalls are listed on each loop guide.

Get started

Build an agent

Guides

Self-host

Output format

Answer tool

Reasoning

Coordinates

Observations

Sampling

Pick an output format

Answer tool

Reasoning

Coordinates in `[0, 1000]`

Observations

Sampling

Common pitfalls

Output format

Answer tool

Reasoning

Coordinates

Observations

Sampling

​Pick an output format

​Answer tool

​Reasoning

​Coordinates in [0, 1000]

​Observations

​Sampling

​Common pitfalls

Pick an output format

Answer tool

Reasoning

Coordinates in `[0, 1000]`

Observations

Sampling

Common pitfalls