Structured outputs

The model is constrained, at the decoding level, to emit a single JSON object matching a schema you provide. Tool calls are fields inside that object, so output is always valid JSON. Works on Holo3.1 (holo3-1-35b-a3b) and Holo3 (holo3-122b-a10b). Read Core concepts first: reasoning, coordinates, observations, and the trim_to_last_n_images helper used below.

Output JSON

Each step, the model emits one object with three fields:

{
  "note": "Submit succeeded; receipt URL is /orders/8421.",
  "thought": "Recording the receipt URL before navigating away.",
  "tool_call": {
    "tool_name": "click",
    "element": "Continue button at the bottom right",
    "x": 932,
    "y": 880
  }
}

string | null

The model’s durable memory: anything from the current screen that future steps will need (URLs, IDs, intermediate answers). null when nothing new is worth recording.

string

required

A one-line plan for the next action.

object

required

One variant of your tool union. Flat: tool_name is a sibling of the arguments, not nested in an args object.

Constrain output to a tool union

Define each tool as a Pydantic model with a Literal[tool_name] field, then use their union as the response schema. The server’s constrained decoder ensures the model emits exactly one variant, and tool_name is the tag you dispatch on at execution time. The example ships three tools (click, write, answer) for illustration. Real agents register a wider toolbox following the same pattern.

from typing import Literal
from pydantic import BaseModel, Field

class ClickArgs(BaseModel):
    """Click at (x, y) coordinates"""
    tool_name: Literal["click"]
    element: str = Field(description="Detailed description of the target UI element to click on")
    x: int = Field(description="X coordinate as integer in [0, 1000]")
    y: int = Field(description="Y coordinate as integer in [0, 1000]")

class WriteArgs(BaseModel):
    """Type text into the currently focused element without clicking first"""
    tool_name: Literal["write"]
    content: str = Field(description="Content to write")
    press_enter: bool = Field(default=False, description="Whether to press Enter after typing")

class AnswerArgs(BaseModel):
    """Provide a final answer"""
    tool_name: Literal["answer"]
    content: str = Field(description="The answer content")

class Step(BaseModel):
    note: str | None = Field(default=None, description="Task-relevant information from the previous observation. Empty if nothing new.")
    thought: str = Field(description="Reasoning about next steps")
    tool_call: ClickArgs | WriteArgs | AnswerArgs

// JSON Schema for the Step union; tool_name is the discriminator you dispatch on
const schema = {
  type: "object",
  properties: {
    note: {
      type: ["string", "null"],
      description: "Task-relevant information from the previous observation. Empty if nothing new.",
    },
    thought: { type: "string", description: "Reasoning about next steps" },
    tool_call: {
      oneOf: [
        {
          type: "object",
          description: "Click at (x, y) coordinates",
          properties: {
            tool_name: { const: "click" },
            element: { type: "string", description: "Detailed description of the target UI element to click on" },
            x: { type: "integer", description: "X coordinate as integer in [0, 1000]" },
            y: { type: "integer", description: "Y coordinate as integer in [0, 1000]" },
          },
          required: ["tool_name", "element", "x", "y"],
        },
        {
          type: "object",
          description: "Type text into the currently focused element without clicking first",
          properties: {
            tool_name: { const: "write" },
            content: { type: "string", description: "Content to write" },
            press_enter: { type: "boolean", description: "Whether to press Enter after typing" },
          },
          required: ["tool_name", "content"],
        },
        {
          type: "object",
          description: "Provide a final answer",
          properties: {
            tool_name: { const: "answer" },
            content: { type: "string", description: "The answer content" },
          },
          required: ["tool_name", "content"],
        },
      ],
    },
  },
  required: ["thought", "tool_call"],
};

Embed the same schema inside the system prompt under an <output_format> block (shown in the loop below). The model was trained with the schema visible in both the prompt and structured_outputs, and dropping either copy noticeably hurts reliability.

Chat layout

Tool results come back as user messages wrapped in <tool_output tool="...">, not as OpenAI tool-role messages:

A complete loop

Pass the schema to structured_outputs and parse content back into your models. Because tool_name is a discriminator, the parsed tool_call narrows to exactly one variant, which is what you dispatch on. Highlighted lines are the ones specific to structured outputs; everything else is the shared loop from Core concepts. Plug in your own screenshot() (browser, OS, emulator) and execute(...) dispatcher.

import json, base64

schema = Step.model_json_schema()
system = render_prompt(tools=...) + f"\n\n<output_format>\n```json\n{json.dumps(schema)}\n```\n</output_format>"

messages = [{"role": "system", "content": system}]

for _ in range(MAX_STEPS):
    image_bytes = screenshot()
    b64 = base64.b64encode(image_bytes).decode()
    messages.append({"role": "user", "content": [
        {"type": "text", "text": "<observation>\n"},
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
        {"type": "text", "text": "\n</observation>"},
    ]})
    trim_to_last_n_images(messages, n=3)

    resp = client.chat.completions.create(
        model=MODEL_NAME,
        messages=messages,
        temperature=0.8,
        extra_body={"structured_outputs": {"json": schema}},
    )
    step = Step.model_validate_json(resp.choices[0].message.content)
    messages.append({"role": "assistant", "content": step.model_dump_json()})

    if step.tool_call.tool_name == "answer":
        return step.tool_call.content

    result = execute(step.tool_call)
    messages.append({
        "role": "user",
        "content": f'<tool_output tool="{step.tool_call.tool_name}">\n{result}\n</tool_output>',
    })

const system =
  renderPrompt() +
  `\n\n<output_format>\n\`\`\`json\n${JSON.stringify(schema)}\n\`\`\`\n</output_format>`;

const messages: any[] = [{ role: "system", content: system }];

for (let i = 0; i < MAX_STEPS; i++) {
  const b64 = (await screenshot()).toString("base64");
  messages.push({
    role: "user",
    content: [
      { type: "text", text: "<observation>\n" },
      { type: "image_url", image_url: { url: `data:image/png;base64,${b64}` } },
      { type: "text", text: "\n</observation>" },
    ],
  });
  trimToLastNImages(messages, 3);

  const resp = await client.chat.completions.create({
    model: MODEL_NAME,
    messages,
    temperature: 0.8,
    ...({ structured_outputs: { json: schema } } as any),
  });
  const step = JSON.parse(resp.choices[0].message.content!);
  messages.push({ role: "assistant", content: JSON.stringify(step) });

  if (step.tool_call.tool_name === "answer") {
    return step.tool_call.content;
  }

  const result = await execute(step.tool_call);
  messages.push({
    role: "user",
    content: `<tool_output tool="${step.tool_call.tool_name}">\n${result}\n</tool_output>`,
  });
}

Format-specific pitfalls

Symptom	Likely cause
Model returns free-form text	`extra_body.structured_outputs.json` is missing, or the schema lacks `Literal[tool_name]` discrimination
Tool result has no effect	Sent as a `tool`-role message instead of a `user` message with a `<tool_output>` wrapper
Model forgets earlier facts	`note` left empty; it is the only memory carried between turns
Reliability drops	Schema present in `structured_outputs` but missing from the `<output_format>` prompt block, or vice versa

Get started

Build an agent

Guides

Self-host

Output JSON

Constrain output to a tool union

Chat layout

A complete loop

Format-specific pitfalls

Next steps

Element localization

API reference

​Output JSON

​Constrain output to a tool union

​Chat layout

​A complete loop

​Format-specific pitfalls

​Next steps

Element localization

API reference

Output JSON

Constrain output to a tool union

Chat layout

A complete loop

Format-specific pitfalls

Next steps