Local inference

Holo4

Both Holo4 models ship in four precisions. Holo4 35B-A3B is under Apache 2.0; Holo4 27B is for research use.

Checkpoint	Holo4 27B	Holo4 35B-A3B	Serve with
BF16	Holo4-27B	Holo4-35B-A3B	vLLM
FP8	Holo4-27B-FP8	Holo4-35B-A3B-FP8	vLLM
NVFP4	Holo4-27B-NVFP4	Holo4-35B-A3B-NVFP4	vLLM on Blackwell
Q4 GGUF	Holo4-27B-GGUF	Holo4-35B-A3B-GGUF	llama.cpp

Recommended vLLM flags, on vLLM 0.28 or newer. Swap the repo and --served-model-name for the other size or precision.

vllm serve Hcompany/Holo4-35B-A3B-FP8 \
  --served-model-name holo4-35b-a3b \
  --max-model-len 262144 \
  --enable-prefix-caching \
  --chat-template-content-format openai \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --limit-mm-per-prompt '{"image": 5, "video": 0}'

Serves on port 8000, so the base URL is http://localhost:8000/v1. The rest of this page covers Holo3.1, with measured memory and speed.

Pick a size

Weights on disk are the minimum GPU or unified memory you need. The KV cache and the image encoder come on top. 35B-A3B is the model the API serves as holo3-1-35b-a3b, and the one to run quantized (next section). 9B and 4B are dense models that fit on a single GPU. 0.8B is for on-device use.

	35B-A3B	9B	4B	0.8B
Weights (BF16)	70.2 GB	18.8 GB	10.4 GB	2.2 GB

Pick a precision

35B-A3B also comes in three quantized checkpoints. NVFP4 needs a third of the memory of BF16.

Checkpoint	Weights	Throughput on DGX Spark	Serve with
BF16	70.2 GB	326 tok/s	vLLM
FP8	37.6 GB	404 tok/s	vLLM
NVFP4 (W4A16)	23.7 GB	568 tok/s	vLLM on Blackwell (DGX Spark, GB10, GB300)
Q4_K_M GGUF	21.3 GB		llama.cpp on Apple Silicon or RTX GPUs

Throughput measured with vLLM’s benchmark tool on an agentic workload: 12k input tokens, 1k output tokens, three 1080p images per request, concurrency 1. NVFP4 uses NVIDIA Model Optimizer in a W4A16 configuration.

Agent speed

Token throughput matters less than how many steps the agent gets through in a minute. Holo3.1 35B-A3B inside HoloDesktop CLI, by hardware and precision: On DGX Spark, NVFP4 plus the harness optimizations cut the average step from 6.8 s to 3.3 s, about twice as fast as the FP8 baseline. The fast harness (holo run --fast) takes one screenshot per step, skips thinking, and sends smaller images. It is quicker and less accurate on fiddly UIs. llama.cpp does not yet cache prefixes or images for the Qwen architecture, which holds its throughput back.

Run it

Start a server

macOS (llama.cpp)
DGX Spark (vLLM)

Use llama.cpp with Metal acceleration and the Q4_K_M GGUF weights. You want an M3 or newer with at least 36 GB of unified memory: the weights alone take 21 GB, and prefix caching needs room for the KV cache. If memory is tight, lower --cache-ram and --ctx-size.

brew install llama.cpp
llama-server -hf Hcompany/Holo-3.1-35B-A3B-GGUF

Tuned flags for higher throughput:

llama-server \
  --hf Hcompany/Holo-3.1-35B-A3B-GGUF \
  --n-gpu-layers 999 \
  --ctx-size 65536 \
  --batch-size 16384 \
  --ubatch-size 2048 \
  --flash-attn 1 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --image-min-tokens 1024 \
  --ctx-checkpoints 8 \
  --cache-ram 32768 \
  --kv-unified \
  --threads 16

Serves on port 8080, so the base URL is http://localhost:8080/v1. Any string works as the model ID.

Use the aarch64 vLLM image with the NVFP4 weights. The GB10’s Blackwell cores run NVFP4 natively.

docker pull vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404
docker run -d --gpus all \
  --shm-size=16g \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404 \
  vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
  --served-model-name holo3-1-35b-a3b \
  --host 0.0.0.0

Tuned flags for higher throughput:

docker run -d --gpus all \
  --shm-size=16g \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404 \
  vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
  --served-model-name holo3-1-35b-a3b \
  --host 0.0.0.0 \
  --gpu-memory-utilization 0.8 \
  --max-model-len 65537 \
  --max-num-batched-tokens 32768 \
  --chat-template-content-format openai \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --limit-mm-per-prompt '{"image": 5, "video": 0}' \
  --mm-encoder-tp-mode data \
  --mm-processor-cache-type shm \
  --mm-processor-cache-gb 15

Serves on port 8000, so the base URL is http://localhost:8000/v1. The model ID you pass must match --served-model-name.

Point your client at it

Same client as the Quickstart, different base_url. The server ignores the API key, but the OpenAI SDKs refuse an empty one.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "http://localhost:8000/v1", apiKey: "local" });

holo run "Open TextEdit and write a short note saying HoloDesktop CLI is installed" \
  --base-url http://localhost:8000/v1 \
  --model holo3-1-35b-a3b

For MCP and ACP hosts, see Local mode from hosts.

Next steps

Core concepts

The harness conventions Holo expects, hosted or local.

HoloDesktop CLI

A ready-made desktop agent that runs on your local server.

Get started

Build an agent

Cookbooks

Guides

Self-host

Holo4

Pick a size

Pick a precision

Agent speed

Run it

Next steps

Core concepts

HoloDesktop CLI

​Holo4

​Pick a size

​Pick a precision

​Agent speed

​Run it

​Next steps

Core concepts

HoloDesktop CLI

Holo4

Pick a size

Pick a precision

Agent speed

Run it

Next steps