> ## Documentation Index
> Fetch the complete documentation index at: https://hub.hcompany.ai/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> H Platform has four products: the Agents API (managed computer-use agents, base URL https://agp.eu.hcompany.ai/api/v2 or https://agp.hcompany.ai/api/v2 for the US), the Models API (OpenAI-compatible Holo vision-language models at https://api.hcompany.ai/v1), HoloDesktop CLI (Holo on the user's own desktop), and HoloTab (a free no-code Chrome extension that runs Holo in the user's browser, with recordable routines and schedules).
> Authenticate with a bearer API key from the HAI_API_KEY environment variable. SDKs: `pip install hai-agents` (Python, `from hai_agents import Client`) and `npm install hai-agents` (TypeScript, `import { HaiAgentsClient } from "hai-agents"`). CLI: `hai`.
> Agents do work in a browser or on a desktop; describe the task as an imperative instruction. To run a task quickly, prefer the pre-built agent `h/web-surfer-flash`. Read results from the session's `latest_answer` after it reaches a terminal status.
> Sessions are the unit of work; wait for a terminal status (completed, failed, timed_out, interrupted) before reading the answer. Use webhooks or the `changes` long-poll endpoint to follow progress.

# Local inference

> Holo4 and Holo3.1 are open weights. Pick a model for your hardware, serve it with llama.cpp or vLLM, and point base_url at your server. Prompts, screenshots, and tool results never leave your network.

Serve the open weights on your own hardware, point `base_url` at your server, and every example in these docs runs as is.

## Pick a setup

| Hardware | Model and precision | Server | Weights |
| :- | :- | :- | -: |
| Mac, M3 or newer, 36 GB or more | Holo3.1 35B-A3B, Q4\_K\_M GGUF | llama.cpp | 21.3 GB |
| DGX Spark, or other Blackwell (GB10, GB300) | Holo3.1 35B-A3B, NVFP4 | vLLM | 23.7 GB |
| NVIDIA GPU server | Holo4 35B-A3B or 27B, FP8 | vLLM 0.28+ | 37.6 GB / 35.9 GB |
| Single 24 GB GPU | Holo3.1 9B, BF16: the most capable dense model on one GPU | vLLM | 18.8 GB |
| Single 16 GB GPU | Holo3.1 4B, BF16: fully local desktop automation with [HoloDesktop CLI](/holo-desktop-cli/introduction) | vLLM | 10.4 GB |
| On-device | Holo3.1 0.8B, BF16: grounding and simple actions | Your on-device runtime | 2.2 GB |

Weights are the minimum GPU or unified memory: the KV cache and image encoder come on top. Holo3.1 35B-A3B is what the API serves as `holo3-1-35b-a3b`. Each Hugging Face model card states its license.

## Start a server

<Tabs>
  <Tab title="macOS (llama.cpp)">
    Use llama.cpp with Metal and the `Q4_K_M` GGUF. Prefix caching needs room for the KV cache on top of the 21 GB of weights. If memory is tight, lower `--cache-ram` and `--ctx-size`.

    ```bash theme={"system"}
    brew install llama.cpp
    llama-server -hf Hcompany/Holo-3.1-35B-A3B-GGUF
    ```

    The server listens on port `8080`: base URL `http://localhost:8080/v1`. Any string works as the model ID. For Holo4, use the [Holo4 GGUF](#holo4-checkpoints) repo instead.

    <Accordion title="Tuned flags for higher throughput">
      ```bash theme={"system"}
      llama-server \
        --hf Hcompany/Holo-3.1-35B-A3B-GGUF \
        --n-gpu-layers 999 \
        --ctx-size 65536 \
        --batch-size 16384 \
        --ubatch-size 2048 \
        --flash-attn 1 \
        --cache-type-k q8_0 \
        --cache-type-v q8_0 \
        --image-min-tokens 1024 \
        --ctx-checkpoints 8 \
        --cache-ram 32768 \
        --kv-unified \
        --threads 16
      ```
    </Accordion>
  </Tab>

  <Tab title="DGX Spark (vLLM)">
    Use the aarch64 vLLM image with NVFP4 weights, which the GB10's Blackwell cores run natively.

    ```bash theme={"system"}
    docker pull vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404
    docker run -d --gpus all \
      --shm-size=16g \
      --network host \
      -v ~/.cache/huggingface:/root/.cache/huggingface \
      vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404 \
      vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
      --served-model-name holo3-1-35b-a3b \
      --host 0.0.0.0
    ```

    The server listens on port `8000`: base URL `http://localhost:8000/v1`. The model ID must match `--served-model-name`. Holo4 needs vLLM 0.28 or newer.

    <Accordion title="Tuned flags for higher throughput">
      ```bash theme={"system"}
      docker run -d --gpus all \
        --shm-size=16g \
        --network host \
        -v ~/.cache/huggingface:/root/.cache/huggingface \
        vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404 \
        vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
        --served-model-name holo3-1-35b-a3b \
        --host 0.0.0.0 \
        --gpu-memory-utilization 0.8 \
        --max-model-len 65537 \
        --max-num-batched-tokens 32768 \
        --chat-template-content-format openai \
        --enable-auto-tool-choice \
        --tool-call-parser qwen3_coder \
        --reasoning-parser qwen3 \
        --limit-mm-per-prompt '{"image": 5, "video": 0}' \
        --mm-encoder-tp-mode data \
        --mm-processor-cache-type shm \
        --mm-processor-cache-gb 15
      ```
    </Accordion>
  </Tab>

  <Tab title="NVIDIA GPU (vLLM, Holo4)">
    These are the recommended flags for vLLM 0.28 or newer. Swap the repo and `--served-model-name` for another [size or precision](#holo4-checkpoints).

    ```bash theme={"system"}
    vllm serve Hcompany/Holo4-35B-A3B-FP8 \
      --served-model-name holo4-35b-a3b \
      --max-model-len 262144 \
      --enable-prefix-caching \
      --chat-template-content-format openai \
      --enable-auto-tool-choice \
      --tool-call-parser qwen3_coder \
      --reasoning-parser qwen3 \
      --limit-mm-per-prompt '{"image": 5, "video": 0}'
    ```

    The server listens on port `8000`: base URL `http://localhost:8000/v1`. The model ID must match `--served-model-name`.
  </Tab>
</Tabs>

## Point your client at it

Use the same client as the [Quickstart](/models-api/quickstart) with a different `base_url`. The examples target vLLM on port `8000`. For llama.cpp, use `8080`. The server ignores the API key, but the OpenAI SDKs refuse an empty one.

<CodeGroup>
  ```bash HoloDesktop CLI theme={"system"}
  holo run "Open TextEdit and write a short note saying HoloDesktop CLI is installed" \
    --base-url http://localhost:8000/v1 \
    --model holo4-35b-a3b
  ```

  ```bash cURL theme={"system"}
  curl http://localhost:8000/v1/models
  ```

  ```python Python theme={"system"}
  from openai import OpenAI

  client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")
  ```

  ```typescript TypeScript theme={"system"}
  import OpenAI from "openai";

  const client = new OpenAI({ baseURL: "http://localhost:8000/v1", apiKey: "local" });
  ```
</CodeGroup>

HoloDesktop CLI's `--model` also sets the prompt format, so pass the Holo version you serve (for example `holo3-1-35b-a3b` for the Holo3.1 recipes). For MCP hosts, see [Run the model yourself](/holo-desktop-cli/getting-started/quickstart#run-the-model-yourself). To run Agents API sessions on this server, see [Run the model on your own GPUs](/agents-api/local-inference).

## Checkpoints

### Holo4 checkpoints

Holo4 35B-A3B is under Apache 2.0. Holo4 27B is for research use (CC BY-NC 4.0). Weight sizes are in parentheses.

| Precision | Holo4 27B | Holo4 35B-A3B | Serve with |
| :- | :- | :- | :- |
| BF16 | [Holo4-27B](https://huggingface.co/Hcompany/Holo4-27B) (54.7 GB) | [Holo4-35B-A3B](https://huggingface.co/Hcompany/Holo4-35B-A3B) (70.2 GB) | vLLM |
| FP8 | [Holo4-27B-FP8](https://huggingface.co/Hcompany/Holo4-27B-FP8) (35.9 GB) | [Holo4-35B-A3B-FP8](https://huggingface.co/Hcompany/Holo4-35B-A3B-FP8) (37.6 GB) | vLLM |
| NVFP4 | [Holo4-27B-NVFP4](https://huggingface.co/Hcompany/Holo4-27B-NVFP4) (22.9 GB) | [Holo4-35B-A3B-NVFP4](https://huggingface.co/Hcompany/Holo4-35B-A3B-NVFP4) (22.5 GB) | vLLM on Blackwell |
| Q4 GGUF | [Holo4-27B-GGUF](https://huggingface.co/Hcompany/Holo4-27B-GGUF) | [Holo4-35B-A3B-GGUF](https://huggingface.co/Hcompany/Holo4-35B-A3B-GGUF) (21.3 GB) | llama.cpp |

### Holo3.1 checkpoints

The 9B, 4B, and 0.8B sizes are not on the API. They are open weights under Apache 2.0.

| Checkpoint | Weights | Throughput on DGX Spark | Serve with |
| :- | -: | -: | :- |
| [35B-A3B BF16](https://huggingface.co/Hcompany/Holo-3.1-35B-A3B) | 70.2 GB | 326 tok/s | vLLM |
| [35B-A3B FP8](https://huggingface.co/Hcompany/Holo-3.1-35B-A3B-FP8) | 37.6 GB | 404 tok/s | vLLM |
| [35B-A3B NVFP4](https://huggingface.co/Hcompany/Holo-3.1-35B-A3B-NVFP4) (W4A16) | 23.7 GB | 568 tok/s | vLLM on Blackwell (DGX Spark, GB10, GB300) |
| [35B-A3B Q4\_K\_M GGUF](https://huggingface.co/Hcompany/Holo-3.1-35B-A3B-GGUF) | 21.3 GB | | llama.cpp on Apple Silicon or RTX GPUs |
| [9B](https://huggingface.co/Hcompany/Holo-3.1-9B) (BF16) | 18.8 GB | | vLLM |
| [4B](https://huggingface.co/Hcompany/Holo-3.1-4B) (BF16) | 10.4 GB | | vLLM |
| [0.8B](https://huggingface.co/Hcompany/Holo-3.1-0.8B) (BF16) | 2.2 GB | | |

Throughput: vLLM benchmark tool, agentic workload (12k input tokens, 1k output tokens, three 1080p images per request, concurrency 1). NVFP4 uses NVIDIA Model Optimizer in a W4A16 configuration.

<Accordion title="Agent speed in HoloDesktop CLI">
  Steps per minute matter more than tokens per second. Model requests per minute for Holo3.1 35B-A3B inside [HoloDesktop CLI](/holo-desktop-cli/introduction):

  | Hardware | Precision, server | Default harness | Fast harness |
  | :- | :- | -: | -: |
  | MacBook M4 Pro | Q4, llama.cpp | 3.6 | 5.5 |
  | DGX Spark | FP8, vLLM | 8.9 | 11.9 |
  | DGX Spark | Q4, llama.cpp | 9.5 | 15.6 |
  | DGX Spark | NVFP4, vLLM | 13.5 | 18.1 |

  On DGX Spark, NVFP4 plus the harness optimizations cut the average step from 6.8 s to 3.3 s, about twice as fast as the FP8 baseline. The fast harness (`holo run --fast`) takes one screenshot per step, skips thinking, and sends smaller images: quicker, less accurate on fiddly UIs. llama.cpp does not yet cache prefixes or images for the Qwen architecture, which holds its throughput back.
</Accordion>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.