> ## Documentation Index
> Fetch the complete documentation index at: https://hub.hcompany.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy via Docker container

The instructions below show you how to serve our models via Docker container.

## Run a vLLM server locally

This section shows you how to install vLLM and run the server locally.

### Pre-requisites

* An NVIDIA GPU with drivers installed

### Installation

1. Install vLLM using the instructions provided by [<u>vLLM</u>](https://docs.vllm.ai/en/latest/getting_started/installation/index.html)
2. Install a compatible version of `transformers`:

```python theme={null}
pip install "transformers<4.53.0"
```

### Example

The snippet below shows you how to run Holo1 3B

```python theme={null}
vllm serve Hcompany/Holo1-3B \
    --dtype bfloat16 \
    --gpu-memory-utilization 0.9 \
    --limit-mm-per-prompt 'image=3,video=0' \
    --mm-processor-kwargs '{"max_pixels": 1003520}' \
    --max-model-len 16384
```

## Deploy via Docker

This section shows you how to deploy our models via Docker container.

### Pre-requisites

* An NVIDIA GPU with drivers installed
* [<u>NVIDIA Container Toolkit</u>](https://github.com/NVIDIA/nvidia-container-toolkit) to allow Docker to access your GPU
* [<u>Docker</u>](https://docs.docker.com/get-docker/) installed and running

### Example

The command below shows you how to run a Holo model.

```python theme={null}
docker run -it --gpus=all --rm -p 8000:8000 vllm/vllm-openai:v0.9.1 \
    --model HCompany/Holo1-3B \
    --dtype bfloat16 \
    --gpu-memory-utilization 0.9 \
    --limit-mm-per-prompt 'image=3,video=0' \
    --mm-processor-kwargs '{"max_pixels": 1003520}' \
    --max-model-len 16384
```

<Info>
  **Keep in mind**

  To run a different Holo model, simply change --model to HCompany/Holo1-7B, for example.
</Info>

## Invoking Holo via API

When the vLLM is running, you can send requests to:

```
http://localhost:8000/v1/chat/completions
```

### Test with curl

```bash theme={null}
curl http://localhost:8000/v1/chat/completions     -H "Content-Type: application/json"     -d '{
        "model": "HCompany/Holo1-3B",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Who won the world series in 2020?"}
        ]
    }'
```

### Test with Python (OpenAI SDK)

1. Install OpenAI client:

```python theme={null}
pip install openai
```

2. Example Python script:

```python theme={null}
from openai import OpenAI

BASE_URL = "http://localhost:8000/v1"
API_KEY = "EMPTY"
MODEL = "HCompany/Holo1-3B"

client = OpenAI(
    base_url=BASE_URL,
    api_key=API_KEY
)

chat_completion = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the world series in 2020?"}
    ]
)

print(chat_completion.choices[0].message.content)
```

<Info>
  **Keep in mind**

  The API key is not used by vLLM but is required by the OpenAI SDK. Use "EMPTY" as a placeholder.
</Info>

### Notes

* `--model` can be set to `HCompany/Holo1-3B` or `# HCompany/Holo1-7B`  or any other applicable Holo model available at the time.
* `--gpus=all` enables all NVIDIA GPUs for the container.
* Our Holo models are multimodal, so you can adjust image/video limits using `--limit-mm-per-prompt`.
* Reduce `--max-model-len` or `--gpu-memory-utilization` if your GPU runs out of memory.
* Ensure your GPU supports bfloat16 (e.g., A100, L40S, RTX 4090, etc.), use float16 otherwise.
* Port 8000 must be free; change it with `-p <host>:8000` if needed.

## Examples

The endpoint is in service. You can use the OpenAI client to perform real-time inference on the deployed Holo model.

* [<u>Using OpenAI Client to invoke Holo1 for a navigation task</u>](https://github.com/hcompai/hai-cookbook/blob/main/holo1/vllm/invoke_navigation.ipynb)
* [<u>Using OpenAI Client to invoke Holo1 for a localization task</u>](https://github.com/hcompai/hai-cookbook/blob/main/holo1/vllm/invoke_localization.ipynb)