Local inference

Serve the open weights on your own hardware, point base_url at your server, and every example in these docs runs as is.

Pick a setup

Hardware	Model and precision	Server	Weights
Mac, M3 or newer, 36 GB or more	Holo3.1 35B-A3B, Q4_K_M GGUF	llama.cpp	21.3 GB
DGX Spark, or other Blackwell (GB10, GB300)	Holo3.1 35B-A3B, NVFP4	vLLM	23.7 GB
NVIDIA GPU server	Holo4 35B-A3B or 27B, FP8	vLLM 0.28+	37.6 GB / 35.9 GB
Single 24 GB GPU	Holo3.1 9B, BF16: the most capable dense model on one GPU	vLLM	18.8 GB
Single 16 GB GPU	Holo3.1 4B, BF16: fully local desktop automation with HoloDesktop CLI	vLLM	10.4 GB
On-device	Holo3.1 0.8B, BF16: grounding and simple actions	Your on-device runtime	2.2 GB

Weights are the minimum GPU or unified memory: the KV cache and image encoder come on top. Holo3.1 35B-A3B is what the API serves as holo3-1-35b-a3b. Each Hugging Face model card states its license.

Start a server

macOS (llama.cpp)
DGX Spark (vLLM)
NVIDIA GPU (vLLM, Holo4)

Use llama.cpp with Metal and the Q4_K_M GGUF. Prefix caching needs room for the KV cache on top of the 21 GB of weights. If memory is tight, lower --cache-ram and --ctx-size.

brew install llama.cpp
llama-server -hf Hcompany/Holo-3.1-35B-A3B-GGUF

The server listens on port 8080: base URL http://localhost:8080/v1. Any string works as the model ID. For Holo4, use the Holo4 GGUF repo instead.

Tuned flags for higher throughput

llama-server \
  --hf Hcompany/Holo-3.1-35B-A3B-GGUF \
  --n-gpu-layers 999 \
  --ctx-size 65536 \
  --batch-size 16384 \
  --ubatch-size 2048 \
  --flash-attn 1 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --image-min-tokens 1024 \
  --ctx-checkpoints 8 \
  --cache-ram 32768 \
  --kv-unified \
  --threads 16

Use the aarch64 vLLM image with NVFP4 weights, which the GB10’s Blackwell cores run natively.

docker pull vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404
docker run -d --gpus all \
  --shm-size=16g \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404 \
  vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
  --served-model-name holo3-1-35b-a3b \
  --host 0.0.0.0

The server listens on port 8000: base URL http://localhost:8000/v1. The model ID must match --served-model-name. Holo4 needs vLLM 0.28 or newer.

Tuned flags for higher throughput

docker run -d --gpus all \
  --shm-size=16g \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404 \
  vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
  --served-model-name holo3-1-35b-a3b \
  --host 0.0.0.0 \
  --gpu-memory-utilization 0.8 \
  --max-model-len 65537 \
  --max-num-batched-tokens 32768 \
  --chat-template-content-format openai \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --limit-mm-per-prompt '{"image": 5, "video": 0}' \
  --mm-encoder-tp-mode data \
  --mm-processor-cache-type shm \
  --mm-processor-cache-gb 15

These are the recommended flags for vLLM 0.28 or newer. Swap the repo and --served-model-name for another size or precision.

vllm serve Hcompany/Holo4-35B-A3B-FP8 \
  --served-model-name holo4-35b-a3b \
  --max-model-len 262144 \
  --enable-prefix-caching \
  --chat-template-content-format openai \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --limit-mm-per-prompt '{"image": 5, "video": 0}'

The server listens on port 8000: base URL http://localhost:8000/v1. The model ID must match --served-model-name.

Point your client at it

Use the same client as the Quickstart with a different base_url. The examples target vLLM on port 8000. For llama.cpp, use 8080. The server ignores the API key, but the OpenAI SDKs refuse an empty one.

holo run "Open TextEdit and write a short note saying HoloDesktop CLI is installed" \
  --base-url http://localhost:8000/v1 \
  --model holo4-35b-a3b

curl http://localhost:8000/v1/models

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "http://localhost:8000/v1", apiKey: "local" });

HoloDesktop CLI’s --model also sets the prompt format, so pass the Holo version you serve (for example holo3-1-35b-a3b for the Holo3.1 recipes). For MCP hosts, see Run the model yourself. To run Agents API sessions on this server, see Run the model on your own GPUs.

Checkpoints

Holo4 checkpoints

Holo4 35B-A3B is under Apache 2.0. Holo4 27B is for research use (CC BY-NC 4.0). Weight sizes are in parentheses.

Precision	Holo4 27B	Holo4 35B-A3B	Serve with
BF16	Holo4-27B (54.7 GB)	Holo4-35B-A3B (70.2 GB)	vLLM
FP8	Holo4-27B-FP8 (35.9 GB)	Holo4-35B-A3B-FP8 (37.6 GB)	vLLM
NVFP4	Holo4-27B-NVFP4 (22.9 GB)	Holo4-35B-A3B-NVFP4 (22.5 GB)	vLLM on Blackwell
Q4 GGUF	Holo4-27B-GGUF	Holo4-35B-A3B-GGUF (21.3 GB)	llama.cpp

Holo3.1 checkpoints

The 9B, 4B, and 0.8B sizes are not on the API. They are open weights under Apache 2.0.

Checkpoint	Weights	Throughput on DGX Spark	Serve with
35B-A3B BF16	70.2 GB	326 tok/s	vLLM
35B-A3B FP8	37.6 GB	404 tok/s	vLLM
35B-A3B NVFP4 (W4A16)	23.7 GB	568 tok/s	vLLM on Blackwell (DGX Spark, GB10, GB300)
35B-A3B Q4_K_M GGUF	21.3 GB		llama.cpp on Apple Silicon or RTX GPUs
9B (BF16)	18.8 GB		vLLM
4B (BF16)	10.4 GB		vLLM
0.8B (BF16)	2.2 GB

Throughput: vLLM benchmark tool, agentic workload (12k input tokens, 1k output tokens, three 1080p images per request, concurrency 1). NVFP4 uses NVIDIA Model Optimizer in a W4A16 configuration.

Agent speed in HoloDesktop CLI

Steps per minute matter more than tokens per second. Model requests per minute for Holo3.1 35B-A3B inside HoloDesktop CLI:

Hardware	Precision, server	Default harness	Fast harness
MacBook M4 Pro	Q4, llama.cpp	3.6	5.5
DGX Spark	FP8, vLLM	8.9	11.9
DGX Spark	Q4, llama.cpp	9.5	15.6
DGX Spark	NVFP4, vLLM	13.5	18.1

On DGX Spark, NVFP4 plus the harness optimizations cut the average step from 6.8 s to 3.3 s, about twice as fast as the FP8 baseline. The fast harness (holo run --fast) takes one screenshot per step, skips thinking, and sends smaller images: quicker, less accurate on fiddly UIs. llama.cpp does not yet cache prefixes or images for the Qwen architecture, which holds its throughput back.

Get started

Build an agent

Cookbooks

Guides

Self-host

Pick a setup

Start a server

Point your client at it

Checkpoints

Holo4 checkpoints

Holo3.1 checkpoints

​Pick a setup

​Start a server

​Point your client at it

​Checkpoints

​Holo4 checkpoints

​Holo3.1 checkpoints

Pick a setup

Start a server

Point your client at it

Checkpoints

Holo4 checkpoints

Holo3.1 checkpoints