NanoVLLM-VoxCPM

Nano-vLLM-VoxCPM is an inference engine for VoxCPM based on Nano-vLLM.

It provides a Python API (sync + async streaming) and an optional FastAPI demo server.

Note

This runtime is GPU-centric (CUDA + Triton + FlashAttention) and does not support CPU-only execution.

Supported VoxCPM Versions

VoxCPM 1.0 (0.5B)

✅ Supported

VoxCPM 1.5

✅ Supported (default)

VoxCPM 2

✅ Supported (voxcpm2 architecture auto-detected from config.json)

Features

  • High throughput on CUDA devices (RTX 4090, H100, A100, etc.)

  • Concurrent requests via an internal scheduler

  • Streaming inference (yield waveform chunks)

  • Optional HTTP demo server (FastAPI)

  • Multi-GPU request-level parallelism via a server pool (one process per GPU)

Prerequisites

  • Linux + NVIDIA GPU (CUDA)

  • Python >= 3.10, < 3.13

  • A local VoxCPM checkpoint directory

Model directory layout

Nano-vLLM-VoxCPM loads weights from *.safetensors files. Your model directory should contain at least:

/path/to/model_dir/
├── config.json
├── audiovae.pth
└── *.safetensors

If your checkpoint only ships weights as .pt / pytorch_model.bin, convert it to safetensors first.

Installation

Install from PyPI:

pip install nano-vllm-voxcpm
# or
uv pip install nano-vllm-voxcpm

Or install from source:

git clone https://github.com/a710128/nanovllm-voxcpm.git
cd nanovllm-voxcpm

# Recommended: use uv + lockfile
uv sync --frozen

Basic usage (Python)

The main entrypoint is nanovllm_voxcpm.VoxCPM.from_pretrained(...).

Generate (async streaming)

If called inside an async event loop, from_pretrained returns an async server pool.

import asyncio
import numpy as np

from nanovllm_voxcpm import VoxCPM


async def main() -> None:
    server = VoxCPM.from_pretrained(
        model="/path/to/model_dir",
        devices=[0],
        max_num_batched_tokens=8192,
        max_num_seqs=16,
        gpu_memory_utilization=0.95,
    )
    await server.wait_for_ready()

    chunks: list[np.ndarray] = []
    async for chunk in server.generate(target_text="Hello world"):
        chunks.append(chunk)  # float32 numpy array

    wav = np.concatenate(chunks, axis=0)
    await server.stop()


if __name__ == "__main__":
    asyncio.run(main())

Generate (sync)

If called outside an event loop, from_pretrained returns a synchronous server pool.

import numpy as np

from nanovllm_voxcpm import VoxCPM


server = VoxCPM.from_pretrained(model="/path/to/model_dir", devices=[0])
chunks = [chunk for chunk in server.generate(target_text="Hello world")]
wav = np.concatenate(chunks, axis=0)
server.stop()

FastAPI demo (HTTP)

The repo includes a FastAPI deployment server in deployment/.

Note

The FastAPI demo service is not published on PyPI. Install it from source as a uv workspace member.

Warning

The FastAPI demo is not production-ready. Do not expose it to untrusted networks without additional security measures.

Install deployment extras

uv sync --package nano-vllm-voxcpm-deployment --frozen

Configure model path

Set the NANOVLLM_MODEL_PATH environment variable (defaults to ~/VoxCPM1.5):

export NANOVLLM_MODEL_PATH=/path/to/model_dir

Start server

uv run fastapi run deployment/app/main.py --host 0.0.0.0 --port 8000

Then open:

  • http://localhost:8000/docs

The /generate endpoint streams audio as MP3:

  • Content-Type: audio/mpeg

  • Payload: streamed MP3 byte stream

Troubleshooting

Missing parameters / weights not found

If you see errors like:

ValueError: Missing parameters: ['base_lm.embed_tokens.weight', ...]

Your model directory likely does not contain *.safetensors weights (some VoxCPM releases ship .pt files). Use a safetensors-converted checkpoint and ensure the *.safetensors files live next to config.json.

Slow startup

The first startup can be slow due to model loading and GPU memory allocation. This is expected.