Local inference

Pick a size

Weights on disk are the minimum GPU or unified memory you need. The KV cache and the image encoder come on top. 35B-A3B is the model the API serves as holo3-1-35b-a3b, and the one to run quantized (next section). 9B and 4B are dense models that fit on a single GPU. 0.8B is for on-device use.

	35B-A3B	9B	4B	0.8B
Weights (BF16)	70.2 GB	18.8 GB	10.4 GB	2.2 GB
Overall	78.3	73.0	72.6	47.5
OSWorld	80.0	71.5	75.8	34.6
Android World	79.3	72.4	72.4
ScreenSpot-Pro	71.5	69.1	66.5	54.3
OSWorld-G	78.8	75.7	73.1	57.5

Overall averages the four H Corporate benchmarks first, then the mean across OSWorld, Android World, H Corporate, ScreenSpot-Pro, and OSWorld-G. Full rows, including the H Corporate categories, are on the Models page.

Pick a precision

35B-A3B also comes in three quantized checkpoints. FP8 and NVFP4 score the same on OSWorld, about 2 points under BF16, and NVFP4 needs a third of the memory.

Checkpoint	Weights	OSWorld	Throughput on DGX Spark	Serve with
BF16	70.2 GB	80.0	326 tok/s	vLLM
FP8	37.6 GB	77.8	404 tok/s	vLLM
NVFP4 (W4A16)	23.7 GB	77.8	568 tok/s	vLLM on Blackwell (DGX Spark, GB10, GB300)
Q4_K_M GGUF	21.3 GB			llama.cpp on Apple Silicon or RTX GPUs

Throughput measured with vLLM’s benchmark tool on an agentic workload: 12k input tokens, 1k output tokens, three 1080p images per request, concurrency 1. NVFP4 uses NVIDIA Model Optimizer in a W4A16 configuration.

Agent speed

Token throughput matters less than how many steps the agent gets through in a minute. Holo3.1 35B-A3B inside HoloDesktop CLI, by hardware and precision: On DGX Spark, NVFP4 plus the harness optimizations cut the average step from 6.8 s to 3.3 s, about twice as fast as the FP8 baseline. The fast harness (holo run --fast) takes one screenshot per step, skips thinking, and sends smaller images. It is quicker and less accurate on fiddly UIs. llama.cpp does not yet cache prefixes or images for the Qwen architecture, which holds its throughput back.

Run it

Start a server

macOS (llama.cpp)
DGX Spark (vLLM)

Use llama.cpp with Metal acceleration and the Q4_K_M GGUF weights. You want an M3 or newer with at least 36 GB of unified memory: the weights alone take 21 GB, and prefix caching needs room for the KV cache. If memory is tight, lower --cache-ram and --ctx-size.

brew install llama.cpp
llama-server -hf Hcompany/Holo-3.1-35B-A3B-GGUF

Tuned flags for higher throughput:

llama-server \
  --hf Hcompany/Holo-3.1-35B-A3B-GGUF \
  --n-gpu-layers 999 \
  --ctx-size 65536 \
  --batch-size 16384 \
  --ubatch-size 2048 \
  --flash-attn 1 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --image-min-tokens 1024 \
  --ctx-checkpoints 8 \
  --cache-ram 32768 \
  --kv-unified \
  --threads 16

Serves on port 8080, so the base URL is http://localhost:8080/v1. Any string works as the model ID.

Use the aarch64 vLLM image with the NVFP4 weights. The GB10’s Blackwell cores run NVFP4 natively.

docker pull vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404
docker run -d --gpus all \
  --shm-size=16g \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404 \
  vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
  --served-model-name holo3-1-35b-a3b \
  --host 0.0.0.0

Tuned flags for higher throughput:

docker run -d --gpus all \
  --shm-size=16g \
  --network host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404 \
  vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
  --served-model-name holo3-1-35b-a3b \
  --host 0.0.0.0 \
  --gpu-memory-utilization 0.8 \
  --max-model-len 65537 \
  --max-num-batched-tokens 32768 \
  --chat-template-content-format openai \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --limit-mm-per-prompt '{"image": 5, "video": 0}' \
  --mm-encoder-tp-mode data \
  --mm-processor-cache-type shm \
  --mm-processor-cache-gb 15

Serves on port 8000, so the base URL is http://localhost:8000/v1. The model ID you pass must match --served-model-name.

Point your client at it

Same client as the Quickstart, different base_url. The server ignores the API key, but the OpenAI SDKs refuse an empty one.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="local")

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "http://localhost:8000/v1", apiKey: "local" });

holo run "Open TextEdit and write a short note saying HoloDesktop CLI is installed" \
  --base-url http://localhost:8000/v1 \
  --model holo3-1-35b-a3b

For MCP and ACP hosts, see Local mode from hosts.

Next steps

Core concepts

The harness conventions Holo expects, hosted or local.

HoloDesktop CLI

A ready-made desktop agent that runs on your local server.

Get started

Build an agent

Guides

Self-host

Pick a size

Pick a precision

Agent speed

Run it

Next steps

Core concepts

HoloDesktop CLI

​Pick a size

​Pick a precision

​Agent speed

​Run it

​Next steps

Core concepts

HoloDesktop CLI

Pick a size

Pick a precision

Agent speed

Run it

Next steps