Structured outputs

Each step, the decoder is constrained to emit one JSON object matching your schema, with the tool call as a field inside it, so output is always valid JSON. Read Core concepts first.

Output JSON

{
  "note": "Submit succeeded; receipt URL is /orders/8421.",
  "thought": "Recording the receipt URL before navigating away.",
  "tool_call": {"tool_name": "click", "element": "Continue button at the bottom right", "x": 932, "y": 880}
}

string | null

The model’s durable memory: facts from the current screen that later steps need (URLs, IDs, partial answers). null when nothing is new.

string

required

A one-line plan for the next action.

object

required

One variant of your tool union. Flat: tool_name sits next to the arguments, not in a nested args object.

Constrain output to a tool union

Define each tool as a Pydantic model with a Literal tool_name, and use their union as the schema. The decoder emits exactly one variant, and you dispatch on tool_name.

Python

from typing import Literal
from pydantic import BaseModel, Field

class ClickArgs(BaseModel):
    """Click at (x, y) coordinates"""
    tool_name: Literal["click"]
    element: str = Field(description="Detailed description of the target UI element to click on")
    x: int = Field(description="X coordinate as integer in [0, 1000]")
    y: int = Field(description="Y coordinate as integer in [0, 1000]")

class WriteArgs(BaseModel):
    """Type text into the currently focused element without clicking first"""
    tool_name: Literal["write"]
    content: str = Field(description="Content to write")
    press_enter: bool = Field(default=False, description="Whether to press Enter after typing")

class AnswerArgs(BaseModel):
    """Provide a final answer"""
    tool_name: Literal["answer"]
    content: str = Field(description="The answer content")

class Step(BaseModel):
    note: str | None = Field(default=None, description="Task-relevant information from the previous observation. Empty if nothing new.")
    thought: str = Field(description="Reasoning about next steps")
    tool_call: ClickArgs | WriteArgs | AnswerArgs

In TypeScript, write the same JSON Schema by hand: a oneOf of tool objects, each with tool_name: { const: "..." }.

Chat layout

Tool results go back as user messages wrapped in <tool_output tool="...">, not as tool messages:

A complete loop

Plug in your own observation() (the <observation> message from Core concepts) and execute(tool_call).

Python

import json

schema = Step.model_json_schema()
system = SYSTEM_PROMPT + f"\n\n<output_format>\n```json\n{json.dumps(schema)}\n```\n</output_format>"
messages = [{"role": "system", "content": system}, {"role": "user", "content": task}]

for _ in range(MAX_STEPS):
    messages.append(observation())
    trim_to_last_n_images(messages, n=3)

    resp = client.chat.completions.create(
        model="holo3-1-35b-a3b",
        messages=messages,
        temperature=0.8,
        extra_body={"structured_outputs": {"json": schema}},
    )
    step = Step.model_validate_json(resp.choices[0].message.content)
    messages.append({"role": "assistant", "content": step.model_dump_json()})

    if step.tool_call.tool_name == "answer":
        return step.tool_call.content
    result = execute(step.tool_call)
    messages.append({"role": "user", "content": f'<tool_output tool="{step.tool_call.tool_name}">\n{result}\n</tool_output>'})

Format-specific pitfalls

Symptom	Likely cause
Model returns free-form text	`structured_outputs.json` missing, or the schema lacks a `Literal` `tool_name`
Tool result has no effect	Sent as a `tool` message instead of a `user` message with a `<tool_output>` wrapper
Model forgets earlier facts	`note` left empty: it is the only memory carried between turns
Reliability drops	Schema in `structured_outputs` but missing from the `<output_format>` block, or the reverse

Get started

Build an agent

Cookbooks

Guides

Self-host

Output JSON

Constrain output to a tool union

Chat layout

A complete loop

Format-specific pitfalls

​Output JSON

​Constrain output to a tool union

​Chat layout

​A complete loop

​Format-specific pitfalls

Output JSON

Constrain output to a tool union

Chat layout

A complete loop

Format-specific pitfalls