The Problem
Most TTS systems rely on discrete tokenization, which loses prosodic detail and makes natural voice cloning difficult. VoxCPM addresses this by modeling speech directly in continuous space, using a diffusion autoregressive architecture to generate speech from text without an intermediate tokenizer. The result is context-aware prosody and zero-shot voice cloning from a short reference clip.
What This Does
VoxCPM is a tokenizer-free TTS system built on a MiniCPM-4 backbone. The architecture lives in src/voxcpm/model/voxcpm.py and is composed of three main modules: an audio VAE (src/voxcpm/modules/audiovae/audiovae.py), a local encoder (src/voxcpm/modules/locenc/localencoder.py), and a diffusion-based decoder (src/voxcpm/modules/locdit/localdit.py). The model uses hierarchical language modeling with FSQ constraints to achieve implicit semantic-acoustic decoupling.
Two model versions exist: VoxCPM-0.5B (640M params, 16kHz audio) and VoxCPM1.5 (800M params, 44.1kHz audio). Both support full fine-tuning and LoRA fine-tuning, with configs in conf/voxcpmv1/ and conf/voxcpmv1.5/. Training scripts are in scripts/trainvoxcpmfinetune.py, with LoRA layer implementations in src/voxcpm/modules/layers/lora.py. A Flask web app (app.py) and CLI (src/voxcpm/cli.py) serve as entry points.
How To Use It
Setup: Install from PyPI with pip install voxcpm. The pyproject.toml declares dependencies but no lockfile exists, so builds are not fully reproducible. Model weights must be downloaded from Hugging Face or ModelScope.
Running it: The README documents a quick start flow. Inference is available through the CLI (src/voxcpm/cli.py) or the Flask app (app.py). For fine-tuning, use the scripts in scripts/ with the YAML configs in conf/ — for example, scripts/trainvoxcpmfinetune.py with conf/voxcpmv1.5/voxcpmfinetunelora.yaml for LoRA training.
Install and run inference via CLI
pip install voxcpm voxcpm --text "Hello, this is a test" --reference examples/example.wav
Fine-tune with LoRA
python scripts/trainvoxcpmfinetune.py --config conf/voxcpmv1.5/voxcpmfinetunelora.yaml
The repo includes a loraftwebui.py for a web-based fine-tuning interface. The examples/traindata_example.jsonl shows the expected training data format.
Real-World Use
A production scenario: a voice assistant platform needs to clone a customer's voice for personalized responses. VoxCPM takes a 5-second reference recording and generates speech matching that voice's timbre, accent, and pacing. The streaming synthesis (RTF ~0.15 on an RTX 4090) allows real-time response generation. Fine-tuning on domain-specific data via the LoRA path lets teams adapt the model without full retraining.
Code Health & Issues
Med - No lockfile: pyproject.toml declares dependencies without pinned versions, making builds non-reproducible across environments. Low - Minimal test coverage: Only 2 test files exist for a 55-file codebase. The model, training, and inference paths are largely untested. Low - CI limited to packaging: The only GitHub Action (publish-to-pypi.yml) handles PyPI releases — no CI runs tests or linting. Low - Fine-tuning is resource-intensive: Full fine-tuning of an 800M-param model requires substantial GPU memory; the LoRA path is the practical option for most users.
The codebase is reasonably organized with clear module separation. The src/voxcpm/training/ package separates config, data, and state management cleanly.
The Bottom Line
VoxCPM is a technically solid, research-grade TTS system with a novel architecture and strong performance claims backed by a technical report. The 1.8M-hour training corpus and two model versions give it real production potential. It's best suited to teams with GPU resources and the ability to handle fine-tuning complexity — not a drop-in solution for lightweight deployments. The missing lockfile and thin test suite are typical of research releases and should be addressed before production use.