> ## Documentation Index
> Fetch the complete documentation index at: https://hub.hcompany.ai/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> H Platform has four products: the Agents API (managed computer-use agents, base URL https://agp.eu.hcompany.ai/api/v2 or https://agp.hcompany.ai/api/v2 for the US), the Models API (OpenAI-compatible Holo vision-language models at https://api.hcompany.ai/v1), HoloDesktop CLI (Holo on the user's own desktop), and HoloTab (a free no-code Chrome extension that runs Holo in the user's browser, with recordable routines and schedules).
> Authenticate with a bearer API key from the HAI_API_KEY environment variable. SDKs: `pip install hai-agents` (Python, `from hai_agents import Client`) and `npm install hai-agents` (TypeScript, `import { HaiAgentsClient } from "hai-agents"`). CLI: `hai`.
> Agents do work in a browser or on a desktop; describe the task as an imperative instruction. To run a task quickly, prefer the pre-built agent `h/web-surfer-flash`. Read results from the session's `latest_answer` after it reaches a terminal status.
> Sessions are the unit of work; wait for a terminal status (completed, failed, timed_out, interrupted) before reading the answer. Use webhooks or the `changes` long-poll endpoint to follow progress.
# Structured outputs
> The Holo3 and Holo3.1 agent loop with one constrained JSON object per step, and the mistakes specific to this format.
export const ChatStrip = ({rows, repeatFrom = 1}) => {
const Body = ({text}) => text.split(/(`[^`]+`)/).map((part, i) => part.startsWith("`") ? {part.slice(1, -1)} : {part});
const Row = ({role, body}) =>
;
};
The model is constrained, at the decoding level, to emit a single JSON object matching a schema you provide. Tool calls are fields inside that object, so output is always valid JSON. Works on Holo3.1 (`holo3-1-35b-a3b`) and Holo3 (`holo3-122b-a10b`).
Use [function calling](/models-api/build-an-agent/function-calling) on Holo3.1 unless you need a constrained JSON object per step.
Read [Core concepts](/models-api/build-an-agent/core-concepts) first: reasoning, coordinates, observations, and the `trim_to_last_n_images` helper used below.
## Output JSON
Each step, the model emits one object with three fields:
```json theme={"system"}
{
"note": "Submit succeeded; receipt URL is /orders/8421.",
"thought": "Recording the receipt URL before navigating away.",
"tool_call": {
"tool_name": "click",
"element": "Continue button at the bottom right",
"x": 932,
"y": 880
}
}
```
The model's durable memory: anything from the current screen that future steps will need (URLs, IDs, intermediate answers). `null` when nothing new is worth recording.
A one-line plan for the next action.
One variant of your tool union. Flat: `tool_name` is a sibling of the arguments, not nested in an `args` object.
## Constrain output to a tool union
Define each tool as a Pydantic model with a `Literal[tool_name]` field, then use their union as the response schema. The server's constrained decoder ensures the model emits exactly one variant, and `tool_name` is the tag you dispatch on at execution time. The example ships three tools (click, write, answer) for illustration. Real agents register a wider toolbox following the same pattern.
```python Python expandable theme={"system"}
from typing import Literal
from pydantic import BaseModel, Field
class ClickArgs(BaseModel):
"""Click at (x, y) coordinates"""
tool_name: Literal["click"]
element: str = Field(description="Detailed description of the target UI element to click on")
x: int = Field(description="X coordinate as integer in [0, 1000]")
y: int = Field(description="Y coordinate as integer in [0, 1000]")
class WriteArgs(BaseModel):
"""Type text into the currently focused element without clicking first"""
tool_name: Literal["write"]
content: str = Field(description="Content to write")
press_enter: bool = Field(default=False, description="Whether to press Enter after typing")
class AnswerArgs(BaseModel):
"""Provide a final answer"""
tool_name: Literal["answer"]
content: str = Field(description="The answer content")
class Step(BaseModel):
note: str | None = Field(default=None, description="Task-relevant information from the previous observation. Empty if nothing new.")
thought: str = Field(description="Reasoning about next steps")
tool_call: ClickArgs | WriteArgs | AnswerArgs
```
```typescript TypeScript expandable theme={"system"}
// JSON Schema for the Step union; tool_name is the discriminator you dispatch on
const schema = {
type: "object",
properties: {
note: {
type: ["string", "null"],
description: "Task-relevant information from the previous observation. Empty if nothing new.",
},
thought: { type: "string", description: "Reasoning about next steps" },
tool_call: {
oneOf: [
{
type: "object",
description: "Click at (x, y) coordinates",
properties: {
tool_name: { const: "click" },
element: { type: "string", description: "Detailed description of the target UI element to click on" },
x: { type: "integer", description: "X coordinate as integer in [0, 1000]" },
y: { type: "integer", description: "Y coordinate as integer in [0, 1000]" },
},
required: ["tool_name", "element", "x", "y"],
},
{
type: "object",
description: "Type text into the currently focused element without clicking first",
properties: {
tool_name: { const: "write" },
content: { type: "string", description: "Content to write" },
press_enter: { type: "boolean", description: "Whether to press Enter after typing" },
},
required: ["tool_name", "content"],
},
{
type: "object",
description: "Provide a final answer",
properties: {
tool_name: { const: "answer" },
content: { type: "string", description: "The answer content" },
},
required: ["tool_name", "content"],
},
],
},
},
required: ["thought", "tool_call"],
};
```
Embed the same schema inside the system prompt under an `` block (shown in [the loop](#a-complete-loop) below). The model was trained with the schema visible in both the prompt and `structured_outputs`, and dropping either copy noticeably hurts reliability.
Use `extra_body={"structured_outputs": {"json": ...}}`, not `tools=[...]` / `tool_choice=...`. The action arrives in `content`, not in a `tool_calls` array.
## Chat layout
Tool results come back as `user` messages wrapped in ``, not as OpenAI `tool`-role messages:
` schema block" },
{ role: "user", body: "`` screenshot and/or text ``" },
{ role: "assistant", body: "`{ note, thought, tool_call }`" },
{ role: "user", body: "`` result ``" },
]}
/>
## A complete loop
Pass the schema to `structured_outputs` and parse `content` back into your models. Because `tool_name` is a discriminator, the parsed `tool_call` narrows to exactly one variant, which is what you dispatch on. Highlighted lines are the ones specific to structured outputs; everything else is the shared loop from [Core concepts](/models-api/build-an-agent/core-concepts). Plug in your own `screenshot()` (browser, OS, emulator) and `execute(...)` dispatcher.
````python Python highlight={3-4,22,24-25,31-34} theme={"system"}
import json, base64
schema = Step.model_json_schema()
system = render_prompt(tools=...) + f"\n\n\n```json\n{json.dumps(schema)}\n```\n"
messages = [{"role": "system", "content": system}]
for _ in range(MAX_STEPS):
image_bytes = screenshot()
b64 = base64.b64encode(image_bytes).decode()
messages.append({"role": "user", "content": [
{"type": "text", "text": "\n"},
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
{"type": "text", "text": "\n"},
]})
trim_to_last_n_images(messages, n=3)
resp = client.chat.completions.create(
model=MODEL_NAME,
messages=messages,
temperature=0.8,
extra_body={"structured_outputs": {"json": schema}},
)
step = Step.model_validate_json(resp.choices[0].message.content)
messages.append({"role": "assistant", "content": step.model_dump_json()})
if step.tool_call.tool_name == "answer":
return step.tool_call.content
result = execute(step.tool_call)
messages.append({
"role": "user",
"content": f'\n{result}\n',
})
````
```typescript TypeScript highlight={1-3,23,25-26,33-36} theme={"system"}
const system =
renderPrompt() +
`\n\n\n\`\`\`json\n${JSON.stringify(schema)}\n\`\`\`\n`;
const messages: any[] = [{ role: "system", content: system }];
for (let i = 0; i < MAX_STEPS; i++) {
const b64 = (await screenshot()).toString("base64");
messages.push({
role: "user",
content: [
{ type: "text", text: "\n" },
{ type: "image_url", image_url: { url: `data:image/png;base64,${b64}` } },
{ type: "text", text: "\n" },
],
});
trimToLastNImages(messages, 3);
const resp = await client.chat.completions.create({
model: MODEL_NAME,
messages,
temperature: 0.8,
...({ structured_outputs: { json: schema } } as any),
});
const step = JSON.parse(resp.choices[0].message.content!);
messages.push({ role: "assistant", content: JSON.stringify(step) });
if (step.tool_call.tool_name === "answer") {
return step.tool_call.content;
}
const result = await execute(step.tool_call);
messages.push({
role: "user",
content: `\n${result}\n`,
});
}
```
## Format-specific pitfalls
| Symptom | Likely cause |
| :--------------------------- | :-------------------------------------------------------------------------------------------------------- |
| Model returns free-form text | `extra_body.structured_outputs.json` is missing, or the schema lacks `Literal[tool_name]` discrimination |
| Tool result has no effect | Sent as a `tool`-role message instead of a `user` message with a `` wrapper |
| Model forgets earlier facts | `note` left empty; it is the only memory carried between turns |
| Reliability drops | Schema present in `structured_outputs` but missing from the `` prompt block, or vice versa |
## Next steps
Get click coordinates from a screenshot.
Endpoint, models, parameters, and limits.