> ## Documentation Index
> Fetch the complete documentation index at: https://hub.hcompany.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run a local model server

> Serve an H Company open-weight model locally with llama.cpp or vLLM, then connect HoloDesktop CLI.

Local mode needs an OpenAI-compatible model server on your own machine. This guide covers llama.cpp on macOS and vLLM on DGX Spark, including the launch flags that matter. Open-weight weights live in the [H Company Hugging Face org](https://huggingface.co/Hcompany).

Holo3 122B is hosted API-only. Use `holo3-122b-a10b` in hosted mode rather than downloading local weights.

Good starting model IDs for local servers:

| Model              | Example model ID                  |
| ------------------ | --------------------------------- |
| Holo 3.1 35B       | `Hcompany/Holo-3.1-35B-A3B`       |
| Holo 3.1 35B GGUF  | `Hcompany/Holo-3.1-35B-A3B-GGUF`  |
| Holo 3.1 35B NVFP4 | `Hcompany/Holo-3.1-35B-A3B-NVFP4` |

<Tabs>
  <Tab title="macOS (llama.cpp)">
    For native performance on macOS with Metal GPU acceleration, use llama.cpp with the open-weight `Q4_K_M` GGUF weights, which balance precision and speed. A MacBook Pro or Max with an M3 chip or newer and at least 36 GB of unified memory is recommended: the `Q4_K_M` weights use roughly 21 GB, and prefix caching improves performance but needs extra memory to pre-allocate the KV cache. Lower `--cache-ram` and `--ctx-size` to reduce memory use.

    Install llama.cpp:

    ```bash theme={null}
    brew install llama.cpp
    ```

    Start the server:

    ```bash theme={null}
    llama-server -hf Hcompany/Holo-3.1-35B-A3B-GGUF
    ```

    For better efficiency, apply these tuned parameters:

    ```bash theme={null}
    llama-server \
      --hf Hcompany/Holo-3.1-35B-A3B-GGUF \
      --n-gpu-layers 999 \
      --ctx-size 65536 \
      --batch-size 16384 \
      --ubatch-size 2048 \
      --flash-attn 1 \
      --cache-type-k q8_0 \
      --cache-type-v q8_0 \
      --image-min-tokens 1024 \
      --ctx-checkpoints 8 \
      --cache-ram 32768 \
      --kv-unified \
      --threads 16
    ```

    llama.cpp serves on port `8080` by default, so the base URL is `http://localhost:8080/v1`. Any string works as the `--model` value.
  </Tab>

  <Tab title="DGX Spark (vLLM)">
    On DGX Spark, use the latest stable vLLM image for aarch64 (`v0.23.0`): `vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404`. The open-weight NVFP4 weights use the Blackwell architecture's NVFP4 support for a good speed and precision tradeoff on GB10 GPUs.

    Pull the image:

    ```bash theme={null}
    docker pull vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404
    ```

    Launch the server:

    ```bash theme={null}
    docker run -d --gpus all \
      --shm-size=16g \
      --network host \
      -v ~/.cache/huggingface:/root/.cache/huggingface \
      vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404 \
      vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
      --served-model-name holo3-1-35b \
      --host 0.0.0.0
    ```

    For better efficiency, apply these tuning parameters:

    ```bash theme={null}
    docker run -d --gpus all \
      --shm-size=16g \
      --network host \
      -v ~/.cache/huggingface:/root/.cache/huggingface \
      vllm/vllm-openai:v0.23.0-aarch64-cu129-ubuntu2404 \
      vllm serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
      --served-model-name holo3-1-35b \
      --host 0.0.0.0 \
      --gpu-memory-utilization 0.8 \
      --max-model-len 65537 \
      --max-num-batched-tokens 32768 \
      --chat-template-content-format openai \
      --enable-auto-tool-choice \
      --tool-call-parser qwen3_coder \
      --reasoning-parser qwen3 \
      --limit-mm-per-prompt '{"image": 5, "video": 0}' \
      --mm-encoder-tp-mode data \
      --mm-processor-cache-type shm \
      --mm-processor-cache-gb 15
    ```

    vLLM serves on port `8000` by default, so the base URL is `http://localhost:8000/v1`. The `--served-model-name` you set (`holo3-1-35b`) is the model ID you must pass to HoloDesktop CLI.
  </Tab>
</Tabs>

## Connect HoloDesktop CLI

With your server running, open a new terminal and point HoloDesktop CLI at it:

<Tabs>
  <Tab title="llama.cpp">
    ```bash theme={null}
    holo run "Open TextEdit and write a short note saying HoloDesktop CLI is installed" \
      --base-url http://localhost:8080/v1 \
      --model holo3-1-35b
    ```
  </Tab>

  <Tab title="vLLM">
    ```bash theme={null}
    holo run "Open TextEdit and write a short note saying HoloDesktop CLI is installed" \
      --base-url http://localhost:8000/v1 \
      --model holo3-1-35b
    ```
  </Tab>
</Tabs>

Local mode does not require `holo login`.

## What's next

Wire local mode into hosts with the environment variables in [Hosted or local models](/holo-desktop-cli/getting-started/hosted-or-local-models#local-mode-from-hosts), then run your first task with the [Quickstart](/holo-desktop-cli/getting-started/quickstart).