Fine-Tuning FAQ¶
General¶
Can I fine-tune for a new language?
Yes. For languages not yet supported by VoxCPM, we recommend full fine-tuning with 500+ hours of target-language data, mixed with some Chinese/English data to reduce forgetting. Use conservative learning rates (1e-5). For the full workflow, see Fine-Tuning Guide.
Training Issues¶
Training takes a long time to reach step 0:
When max_batch_tokens is enabled, the training script needs the duration of every target and reference audio file before filtering long samples. Missing duration or ref_duration values are obtained from audio file headers during startup.
Set
preprocessing_num_workersto match the available CPU and storage bandwidth. The default configs use8.For the fastest startup on very large datasets, write numeric
durationandref_durationvalues, in seconds, into the JSONL manifest.num_workersonly controls the training DataLoader; it does not control manifest preprocessing.Setting
max_batch_tokens: 0skips duration-based filtering, but unusually long samples may then cause OOM errors.On network storage, too many workers can increase metadata contention. Try
4or8and measure on the actual filesystem.
Out of memory (OOM):
Reduce
batch_sizeormax_batch_tokensto filter out long samples.Increase
grad_accum_stepsto maintain effective batch size with less per-step memory.Switch to LoRA fine-tuning — it uses roughly half the VRAM of full fine-tuning.
For multi-GPU (DDP), expect ~10 GB additional VRAM per card from gradient buckets and NCCL buffers.
Training loss not converging:
Decrease
learning_rate.Increase
warmup_steps.Check data quality — noisy audio or mismatched transcripts are common culprits.
Resume training shows wrong step count: (#187)
This is a known bug in multi-GPU training. Ensure you are using the latest version of the training scripts.
Output Quality Issues¶
Model ignores input text after fine-tuning (overfitting): (#169)
The model has overfit to reproducing training audio without text conditioning. This is the most common fine-tuning failure mode and can emerge surprisingly early (within a few hundred steps for small datasets).
Keep
training_cfg_rate=0.1(do NOT set it to 0).Keep
weight_decay=0.01.Reduce learning rate to
1e-5(full FT) or1e-4(LoRA).Monitor checkpoints at each
save_interval. For most single-speaker tasks, 1–3 epochs is sufficient — training beyond that often hurts.
Generation doesn’t stop (runaway output): (#195, #124)
Check your training data for clips with long trailing silence (>0.5 s) and trim them — this is the most common cause.
Enable
retry_badcase=Trueat inference time as a safety net.If fine-tuning a new language, the stop loss and diffusion loss may converge at different rates — try increasing the stop loss weight (
lambdas > loss/stop).
LoRA Issues¶
Poor LoRA quality:
Increase
r(LoRA rank) — try 32 or 64 for harder tasks like style or language adaptation.Adjust
alpha— tryalpha = roralpha = 2*r.Ensure
enable_dit: true— this is essential for voice quality.Increase training steps if the model has not converged yet.
LoRA not taking effect at inference:
Ensure the inference LoRA config (
r,alpha,enable_lm,enable_dit) matches the training config exactly.Check the return value of
load_lora—skipped_keysshould be empty.Verify
set_lora_enabled(True)is called if you previously disabled it.
Checkpoint Issues¶
Checkpoint loading errors:
Full fine-tuning: the checkpoint directory must contain
model.safetensors(orpytorch_model.bin),config.json, andaudiovae.pth.LoRA: the checkpoint directory must contain
lora_weights.safetensors(orlora_weights.ckpt) andlora_config.json.