Function calling

Holo4 (holo4-27b, holo4-35b-a3b) and Holo3.1 (holo3-1-35b-a3b) speak the standard OpenAI tool-calling API, so they plug into agent frameworks that already do. Read Core concepts first: reasoning, coordinates, observations, and the trim_to_last_n_images helper used below.

Declare tools

Pass tool schemas via tools, and set tool_choice="required" so the model acts on every step. Do not put tool-format examples in the system prompt. The model renders the call in its own native format from tools alone, and a conflicting example degrades quality. The example ships two tools (click, answer) for illustration. Real agents register a wider toolbox the same way, and the tools need not touch a screen: see Mix screen, code, and MCP tools.

tools = [
    {
        "type": "function",
        "function": {
            "name": "click",
            "description": "Click at (x, y) coordinates",
            "parameters": {
                "type": "object",
                "properties": {
                    "element": {"type": "string", "description": "Detailed description of the target UI element"},
                    "x": {"type": "integer", "description": "X coordinate as integer in [0, 1000]"},
                    "y": {"type": "integer", "description": "Y coordinate as integer in [0, 1000]"},
                },
                "required": ["element", "x", "y"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "answer",
            "description": "Provide a final answer",
            "parameters": {
                "type": "object",
                "properties": {"content": {"type": "string", "description": "The answer content"}},
                "required": ["content"],
            },
        },
    },
]

const tools = [
  {
    type: "function",
    function: {
      name: "click",
      description: "Click at (x, y) coordinates",
      parameters: {
        type: "object",
        properties: {
          element: { type: "string", description: "Detailed description of the target UI element" },
          x: { type: "integer", description: "X coordinate as integer in [0, 1000]" },
          y: { type: "integer", description: "Y coordinate as integer in [0, 1000]" },
        },
        required: ["element", "x", "y"],
      },
    },
  },
  {
    type: "function",
    function: {
      name: "answer",
      description: "Provide a final answer",
      parameters: {
        type: "object",
        properties: { content: { type: "string", description: "The answer content" } },
        required: ["content"],
      },
    },
  },
] as const;

Chat layout

Tool results go back as tool-role messages keyed by tool_call_id:

A complete loop

The action comes back in message.tool_calls; each call carries a function.name, a JSON function.arguments string, and a unique id. There is no note field in this format: anything the model must carry across turns goes in the assistant content. Highlighted lines are the ones specific to function calling; everything else is the shared loop from Core concepts. Plug in your own screenshot() (browser, OS, emulator) and execute(name, args) dispatcher.

import json, base64

messages = [{"role": "system", "content": render_prompt()}]

for _ in range(MAX_STEPS):
    image_bytes = screenshot()
    b64 = base64.b64encode(image_bytes).decode()
    messages.append({"role": "user", "content": [
        {"type": "text", "text": "<observation>\n"},
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
        {"type": "text", "text": "\n</observation>"},
    ]})
    trim_to_last_n_images(messages, n=3)

    resp = client.chat.completions.create(
        model=MODEL_NAME,
        messages=messages,
        tools=tools,
        tool_choice="required",
        temperature=0.8,
    )
    msg = resp.choices[0].message
    messages.append(msg)

    call = msg.tool_calls[0]
    args = json.loads(call.function.arguments)

    if call.function.name == "answer":
        return args["content"]

    result = execute(call.function.name, args)
    messages.append({
        "role": "tool",
        "tool_call_id": call.id,
        "content": str(result),
    })

const messages: any[] = [{ role: "system", content: renderPrompt() }];

for (let i = 0; i < MAX_STEPS; i++) {
  const b64 = (await screenshot()).toString("base64");
  messages.push({
    role: "user",
    content: [
      { type: "text", text: "<observation>\n" },
      { type: "image_url", image_url: { url: `data:image/png;base64,${b64}` } },
      { type: "text", text: "\n</observation>" },
    ],
  });
  trimToLastNImages(messages, 3);

  const resp = await client.chat.completions.create({
    model: MODEL_NAME,
    messages,
    tools,
    tool_choice: "required",
    temperature: 0.8,
  });
  const msg = resp.choices[0].message;
  messages.push(msg);

  const call = msg.tool_calls![0];
  const args = JSON.parse(call.function.arguments);

  if (call.function.name === "answer") {
    return args.content;
  }

  const result = await execute(call.function.name, args);
  messages.push({
    role: "tool",
    tool_call_id: call.id,
    content: String(result),
  });
}

Mix screen, code, and MCP tools

Holo4 is trained on tasks that cross interfaces, so one toolbox can hold screen actions, a code runner, and the tools of an MCP server. The model picks one per step: it reads a page with click, then switches to run_python for the arithmetic. Nothing else in the loop changes. MCP tools already carry a JSON schema, so they convert one to one:

Python

def to_openai_tool(tool):
    return {
        "type": "function",
        "function": {"name": tool.name, "description": tool.description or "", "parameters": tool.input_schema},
    }

mcp_tools = (await session.list_tools()).tools
tools = [click, type_text, run_python] + [to_openai_tool(t) for t in mcp_tools] + [answer]

Route each call to where it runs: screen actions to your browser or OS driver, run_python to a sandbox, and MCP tool names to session.call_tool(name, args). Screenshots stay under the same 3-image budget whichever tool ran.

Browser agent

Playwright, screenshots in, clicks out.

Code agent

A Python runner as the only tool.

MCP agent

Any MCP server’s tools, passed as functions.

Hybrid agent

Browser and code in one loop.

Format-specific pitfalls

Symptom	Likely cause
Tool calls come back as plain text	`tool_choice` not set to `required`, or the system prompt contains conflicting tool-format examples
Tool result ignored	Sent as a `user` message instead of a `tool`-role message with a matching `tool_call_id`
Model forgets earlier facts	Nothing written to the assistant `content`

Get started

Build an agent

Cookbooks

Guides

Self-host

Declare tools

Chat layout

A complete loop

Mix screen, code, and MCP tools

Browser agent

Code agent

MCP agent

Hybrid agent

Format-specific pitfalls

Next steps

Element localization

API reference

​Declare tools

​Chat layout

​A complete loop

​Mix screen, code, and MCP tools

Browser agent

Code agent

MCP agent

Hybrid agent

​Format-specific pitfalls

​Next steps

Element localization

API reference

Declare tools

Chat layout

A complete loop

Mix screen, code, and MCP tools

Format-specific pitfalls

Next steps