> ## Documentation Index
> Fetch the complete documentation index at: https://hub.hcompany.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> H Platform has four products: the Agents API (managed computer-use agents, base URL https://agp.eu.hcompany.ai/api/v2 or https://agp.hcompany.ai/api/v2 for the US), the Models API (OpenAI-compatible Holo vision-language models at https://api.hcompany.ai/v1), HoloDesktop CLI (Holo on the user's own desktop), and HoloTab (a free no-code Chrome extension that runs Holo in the user's browser, with recordable routines and schedules).
> Authenticate with a bearer API key from the HAI_API_KEY environment variable. SDKs: `pip install hai-agents` (Python, `from hai_agents import Client`) and `npm install hai-agents` (TypeScript, `import { HaiAgentsClient } from "hai-agents"`). CLI: `hai`.
> Agents do work in a browser or on a desktop; describe the task as an imperative instruction. To run a task quickly, prefer the pre-built agent `h/web-surfer-flash`. Read results from the session's `latest_answer` after it reaches a terminal status.
> Sessions are the unit of work; wait for a terminal status (completed, failed, timed_out, interrupted) before reading the answer. Use webhooks or the `changes` long-poll endpoint to follow progress.

# Local inference

> Holo3.1 is open weights under Apache 2.0. Serve it with llama.cpp or vLLM, point base_url at your server, and every example in these docs runs as is. Screenshots and prompts never leave your network.

export const LocalSpeedChart = () => {
  const groups = [{
    label: "Default harness",
    bars: [{
      label: "Q4",
      sub: "llama.cpp",
      device: "mac",
      value: 3.6
    }, {
      label: "FP8",
      sub: "vLLM",
      device: "spark",
      value: 8.9
    }, {
      label: "Q4",
      sub: "llama.cpp",
      device: "spark",
      value: 9.5
    }, {
      label: "NVFP4",
      sub: "vLLM",
      device: "spark",
      value: 13.5
    }]
  }, {
    label: "Fast harness",
    bars: [{
      label: "Q4",
      sub: "llama.cpp",
      device: "mac",
      value: 5.5
    }, {
      label: "FP8",
      sub: "vLLM",
      device: "spark",
      value: 11.9
    }, {
      label: "Q4",
      sub: "llama.cpp",
      device: "spark",
      value: 15.6
    }, {
      label: "NVFP4",
      sub: "vLLM",
      device: "spark",
      value: 18.1
    }]
  }];
  const W = 760;
  const H = 330;
  const left = 40;
  const right = 16;
  const top = 44;
  const plotH = 200;
  const max = 20;
  const barW = 54;
  const gap = 18;
  const groupW = 4 * barW + 3 * gap;
  const groupGap = W - left - right - 2 * groupW;
  const y = v => top + plotH - v / max * plotH;
  const mono = {
    fontFamily: "ui-monospace, SFMono-Regular, Menlo, monospace"
  };
  const Legend = ({x, dark, label}) => <g>
      {dark ? <rect x={x} y={top - 34} width="12" height="12" rx="3" className="text-zinc-900 dark:text-zinc-100" fill="currentColor" /> : <rect x={x} y={top - 34} width="12" height="12" rx="3" className="text-zinc-400 dark:text-zinc-500" fill="none" stroke="currentColor" strokeWidth="1.5" />}
      <text x={x + 18} y={top - 24} fontSize="12" className="text-zinc-600 dark:text-zinc-400" fill="currentColor">{label}</text>
    </g>;
  return <div className="not-prose my-8 w-full">
      <svg viewBox={`0 0 ${W} ${H}`} width="100%" className="block" role="img" aria-label="Model requests per minute for Holo3.1 35B-A3B by hardware, precision, and harness">
        <text x={left} y={top - 24} fontSize="12" className="text-zinc-500 dark:text-zinc-400" fill="currentColor">Model requests per minute</text>
        <Legend x={W - right - 250} dark label="DGX Spark" />
        <Legend x={W - right - 130} label="MacBook M4 Pro" />

        <g className="text-zinc-200 dark:text-zinc-800" stroke="currentColor" strokeWidth="1">
          {[5, 10, 15, 20].map(v => <line key={v} x1={left} y1={y(v)} x2={W - right} y2={y(v)} strokeDasharray="2 4" />)}
          <line x1={left} y1={y(0)} x2={W - right} y2={y(0)} />
        </g>
        <g className="text-zinc-400 dark:text-zinc-500" fill="currentColor" fontSize="11" style={mono}>
          {[5, 10, 15, 20].map(v => <text key={v} x={left - 8} y={y(v) + 4} textAnchor="end">{v}</text>)}
        </g>

        {groups.map((g, gi) => {
    const gx = left + gi * (groupW + groupGap);
    return <g key={g.label}>
              {g.bars.map((b, bi) => {
      const x = gx + bi * (barW + gap);
      const spark = b.device === "spark";
      const barTop = y(b.value);
      const h = y(0) - barTop;
      return <g key={bi}>
                    {spark ? <path d={`M${x} ${y(0)} V${barTop + 4} a4 4 0 0 1 4 -4 h${barW - 8} a4 4 0 0 1 4 4 V${y(0)} Z`} className="text-zinc-900 dark:text-zinc-100" fill="currentColor" /> : <path d={`M${x + 0.75} ${y(0)} V${barTop + 4} a4 4 0 0 1 4 -4 h${barW - 9.5} a4 4 0 0 1 4 4 V${y(0)}`} className="text-zinc-400 dark:text-zinc-500" fill="none" stroke="currentColor" strokeWidth="1.5" />}
                    <text x={x + barW / 2} y={barTop - 8} textAnchor="middle" fontSize="13" fontWeight="600" className="text-zinc-900 dark:text-zinc-100" fill="currentColor" style={mono}>{b.value}</text>
                    <text x={x + barW / 2} y={y(0) + 18} textAnchor="middle" fontSize="12" fontWeight="500" className="text-zinc-800 dark:text-zinc-200" fill="currentColor" style={mono}>{b.label}</text>
                    <text x={x + barW / 2} y={y(0) + 32} textAnchor="middle" fontSize="11" className="text-zinc-500 dark:text-zinc-400" fill="currentColor">{b.sub}</text>
                  </g>;
    })}
              <text x={gx + groupW / 2} y={y(0) + 60} textAnchor="middle" fontSize="13" fontWeight="600" className="text-zinc-900 dark:text-zinc-100" fill="currentColor">{g.label}</text>
            </g>;
  })}
      </svg>
    </div>;
};

## Pick a size

Weights on disk are the minimum GPU or unified memory you need. The KV cache and the image encoder come on top. 35B-A3B is the model the API serves as `holo3-1-35b-a3b`, and the one to run quantized (next section). 9B and 4B are dense models that fit on a single GPU. 0.8B is for on-device use.

|                | [35B-A3B](https://huggingface.co/Hcompany/Holo-3.1-35B-A3B) | [9B](https://huggingface.co/Hcompany/Holo-3.1-9B) | [4B](https://huggingface.co/Hcompany/Holo-3.1-4B) | [0.8B](https://huggingface.co/Hcompany/Holo-3.1-0.8B) |
| :------------- | ----------------------------------------------------------: | ------------------------------------------------: | ------------------------------------------------: | ----------------------------------------------------: |
| Weights (BF16) |                                                     70.2 GB |                                           18.8 GB |                                           10.4 GB |                                                2.2 GB |
| **Overall**    |                                                    **78.3** |                                              73.0 |                                              72.6 |                                                  47.5 |
| OSWorld        |                                                    **80.0** |                                              71.5 |                                              75.8 |                                                  34.6 |
| Android World  |                                                    **79.3** |                                              72.4 |                                              72.4 |                                                       |
| ScreenSpot-Pro |                                                    **71.5** |                                              69.1 |                                              66.5 |                                                  54.3 |
| OSWorld-G      |                                                    **78.8** |                                              75.7 |                                              73.1 |                                                  57.5 |

Overall averages the four H Corporate benchmarks first, then the mean across OSWorld, Android World, H Corporate, ScreenSpot-Pro, and OSWorld-G. Full rows, including the H Corporate categories, are on the [Models](/models-api/introduction#benchmarks) page.

## Pick a precision

35B-A3B also comes in three quantized checkpoints. FP8 and NVFP4 score the same on OSWorld, about 2 points under BF16, and NVFP4 needs a third of the memory.

| Checkpoint                                                              | Weights | OSWorld | Throughput on DGX Spark | Serve with                                 |
| :---------------------------------------------------------------------- | ------: | ------: | ----------------------: | :----------------------------------------- |
| [BF16](https://huggingface.co/Hcompany/Holo-3.1-35B-A3B)                | 70.2 GB |    80.0 |               326 tok/s | vLLM                                       |
| [FP8](https://huggingface.co/Hcompany/Holo-3.1-35B-A3B-FP8)             | 37.6 GB |    77.8 |               404 tok/s | vLLM                                       |
| [NVFP4](https://huggingface.co/Hcompany/Holo-3.1-35B-A3B-NVFP4) (W4A16) | 23.7 GB |    77.8 |               568 tok/s | vLLM on Blackwell (DGX Spark, GB10, GB300) |
| [Q4\_K\_M GGUF](https://huggingface.co/Hcompany/Holo-3.1-35B-A3B-GGUF)  | 21.3 GB |         |                         | llama.cpp on Apple Silicon or RTX GPUs     |

Throughput measured with vLLM's benchmark tool on an agentic workload: 12k input tokens, 1k output tokens, three 1080p images per request, concurrency 1. NVFP4 uses NVIDIA Model Optimizer in a W4A16 configuration.

## Agent speed

Token throughput matters less than how many steps the agent gets through in a minute. Holo3.1 35B-A3B inside [HoloDesktop CLI](/holo-desktop-cli/introduction), by hardware and precision:

<LocalSpeedChart />

On DGX Spark, NVFP4 plus the harness optimizations cut the average step from 6.8 s to 3.3 s, about twice as fast as the FP8 baseline. The fast harness (`holo run --fast`) takes one screenshot per step, skips thinking, and sends smaller images. It is quicker and less accurate on fiddly UIs. llama.cpp does not yet cache prefixes or images for the Qwen architecture, which holds its throughput back.

## Run it

<Steps>
  <Step title="Start a server">
    <Tabs>
      <Tab title="macOS (llama.cpp)">
        Use llama.cpp with Metal acceleration and the `Q4_K_M` GGUF weights. You want an M3 or newer with at least 36 GB of unified memory: the weights alone take 21 GB, and prefix caching needs room for the KV cache. If memory is tight, lower `--cache-ram` and `--ctx-size`.

        ```bash theme={"system"}
        brew install llama.cpp
        llama-server -hf Hcompany/Holo-3.1-35B-A3B-GGUF
        ```

        Tuned flags for higher throughput:

        ```bash expandable theme={"system"}
        llama-server \
          --hf Hcompany/Holo-3.1-35B-A3B-GGUF \
          --n-gpu-layers 999 \
          --ctx-size 65536 \
          --batch-size 16384 \
          --ubatch-size 2048 \
          --flash-attn 1 \
          --cache-type-k q8_0 \
          --cache-type-v q8_0 \
          --image-min-tokens 1024 \
          --ctx-checkpoints 8 \
          --cache-ram 32768 \
          --kv-unified \
          --threads 16
        ```

        Serves on port `8080`, so the base URL is `http://localhost:8080/v1`. Any string works as the model ID.
      </Tab>

      <Tab title="DGX Spark (vLLM)">
        Use the aarch64 vLLM image with the NVFP4 weights. The GB10's Blackwell cores run NVFP4 natively.

        ```bash theme={"system"}
        docker pull vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404
        docker run -d --gpus all \
          --shm-size=16g \
          --network host \
          -v ~/.cache/huggingface:/root/.cache/huggingface \
          vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404 \
          vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
          --served-model-name holo3-1-35b-a3b \
          --host 0.0.0.0
        ```

        Tuned flags for higher throughput:

        ```bash expandable theme={"system"}
        docker run -d --gpus all \
          --shm-size=16g \
          --network host \
          -v ~/.cache/huggingface:/root/.cache/huggingface \
          vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404 \
          vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
          --served-model-name holo3-1-35b-a3b \
          --host 0.0.0.0 \
          --gpu-memory-utilization 0.8 \
          --max-model-len 65537 \
          --max-num-batched-tokens 32768 \
          --chat-template-content-format openai \
          --enable-auto-tool-choice \
          --tool-call-parser qwen3_coder \
          --reasoning-parser qwen3 \
          --limit-mm-per-prompt '{"image": 5, "video": 0}' \
          --mm-encoder-tp-mode data \
          --mm-processor-cache-type shm \
          --mm-processor-cache-gb 15
        ```

        Serves on port `8000`, so the base URL is `http://localhost:8000/v1`. The model ID you pass must match `--served-model-name`.
      </Tab>
    </Tabs>
  </Step>

  <Step title="Point your client at it">
    Same client as the [Quickstart](/models-api/quickstart), different `base_url`. The server ignores the API key, but the OpenAI SDKs refuse an empty one.

    <CodeGroup>
      ```python Python theme={"system"}
      from openai import OpenAI

      client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")
      ```

      ```typescript TypeScript theme={"system"}
      import OpenAI from "openai";

      const client = new OpenAI({ baseURL: "http://localhost:8000/v1", apiKey: "local" });
      ```

      ```bash HoloDesktop CLI theme={"system"}
      holo run "Open TextEdit and write a short note saying HoloDesktop CLI is installed" \
        --base-url http://localhost:8000/v1 \
        --model holo3-1-35b-a3b
      ```
    </CodeGroup>

    For MCP and ACP hosts, see [Local mode from hosts](/holo-desktop-cli/getting-started/hosted-or-local-models#local-mode-from-hosts).
  </Step>
</Steps>

## Next steps

<CardGroup cols={2}>
  <Card title="Core concepts" icon="brain" href="/models-api/build-an-agent/core-concepts">
    The harness conventions Holo expects, hosted or local.
  </Card>

  <Card title="HoloDesktop CLI" icon="desktop" href="/holo-desktop-cli/getting-started/hosted-or-local-models">
    A ready-made desktop agent that runs on your local server.
  </Card>
</CardGroup>