Architecture

VoxCPM adopts a tokenizer-free, diffusion autoregressive architecture that models speech in continuous latent space rather than discrete tokens. This page first explains the high-level design shared across the VoxCPM family, then summarizes the main architectural improvements introduced in VoxCPM 2.

VoxCPM Model Architecture

High-level VoxCPM pipeline shared across the model family.

Overview

Unlike mainstream TTS approaches that convert speech into discrete tokens, VoxCPM uses an end-to-end architecture that directly generates continuous speech representations from text. Built on a MiniCPM-4 backbone, the system uses hierarchical language modeling and FSQ-constrained continuous latents to separate high-level semantic planning from low-level acoustic rendering.

Across VoxCPM 1.0, 1.5, and 2, the family shares the same high-level generation path:

text -> four-stage generative pipeline -> AudioVAE decoder -> waveform

The generative backbone consists of four stages:

#

Stage

Description

1

Local Encoder

Encodes input audio patches into compact local representations. Groups consecutive audio frames into patches to reduce the effective sequence length for the language model.

2

Text-Semantic LM

A causal language model (based on MiniCPM-4) that jointly processes text tokens and audio embeddings to capture high-level semantic intent. This stage handles the “what to say” — planning prosody, pacing, and emphasis from the text content.

3

Residual Acoustic LM

Fuses semantic-level and acoustic-level information to model fine-grained acoustic details. Bridges the gap between the text-semantic planning and the final audio generation.

4

Local DiT (CFM)

A Conditional Flow Matching (CFM) diffusion transformer that generates continuous audio latents conditioned on the LM outputs. Produces high-fidelity speech patches at each autoregressive step.

The generated latents are then decoded by AudioVAE into raw waveforms. AudioVAE is a supporting codec component used around the generative backbone: it provides latent representations during training and converts predicted latents back to waveform samples during inference.

Across versions, the codec layer evolves from a symmetric audio VAE in VoxCPM 1.x to AudioVAE V2 in VoxCPM 2, which uses asymmetric 16kHz encode -> 48kHz decode and sample-rate conditioning.

Three design choices define the shared VoxCPM architecture:

  • Tokenizer-free continuous modeling preserves fine-grained acoustic detail instead of compressing speech into discrete tokens.

  • Hierarchical semantic-acoustic separation lets the model split high-level planning from low-level rendering.

  • Patch-level autoregressive generation reduces sequence length and helps the system scale to longer and faster synthesis settings.

For version-by-version model selection and migration guidance, see Version History.

VoxCPM 2 Improvements

VoxCPM 2 keeps the same overall four-stage structure, but redesigns several internal information pathways. These changes are important because they explain why VoxCPM 2 improves expressiveness, controllability, and output quality without changing the high-level mental model of the system.

Area

VoxCPM 1.x

VoxCPM 2

Residual Acoustic LM fusion

Additive fusion

Concat + projection fusion for richer semantic-acoustic mixing

Local DiT conditioning

Single fused conditioning token

Multi-token conditioning prefix to preserve more information

Reference audio pathway

Prompt continuation only

Structurally isolated reference-audio channel

AudioVAE

Symmetric encode/decode

AudioVAE V2 with asymmetric 16kHz encode -> 48kHz decode

Residual Acoustic LM Fusion

In VoxCPM 1.x, the Residual Acoustic LM combines the semantic LM output and local acoustic features through addition. VoxCPM 2 replaces this with concatenation followed by a learnable projection.

This gives the model more freedom to decide how semantic intent and acoustic evidence should interact, instead of forcing them into the same representation through element-wise addition. In practice, this supports richer acoustic detail and stronger expressiveness.

Local DiT Conditioning

The Local DiT is a diffusion transformer operating within each audio patch. Instead of using a single early-fused conditioning signal, VoxCPM 2 feeds the DiT a multi-token conditioning prefix derived from the semantic and acoustic pathways.

This preserves more information for attention to work with inside the DiT, reducing information collapse from premature fusion. The result is a more expressive and controllable final acoustic generation stage.

Isolated Reference Audio Channel

VoxCPM 1.x primarily supports voice cloning through prompt continuation. VoxCPM 2 adds a structurally isolated reference-audio pathway, separating timbre reference from continuation context.

This architectural change is what enables stronger reference-only cloning behavior and makes it easier to combine voice identity control with other generation modes.

Why AudioVAE V2 Matters

AudioVAE V2 is not just a higher-sample-rate decoder. Its asymmetric encode/decode design keeps the language-model-side sequence efficient while still producing 48kHz output directly.

This is a key architectural improvement because it raises output fidelity without requiring a proportional increase in sequence length or a separate upsampling stage.

Where to Go Next

  • For release-by-release comparisons and migration guidance, see Version History.

  • For VoxCPM 2 specific details and examples, see VoxCPM 2.

References