The Problem

Existing TTS models force a triage between voice cloning quality, generation speed, and VRAM footprint. Most high-quality systems require 10GB+ VRAM and realtime-or-slower inference, while lightweight models cap at 24khz audio and lower cloning fidelity. This repository addresses the gap of achieving SOTA voice cloning quality competitive with models 10x larger, at 150x realtime speed, constrained to 1GB VRAM with 48khz output.

What This Does

The repository implements a zipvoice-based TTS architecture distributed across the zipvoice/ directory (36 files). The core model logic resides in zipvoice/models/zipvoice.py and zipvoice/models/zipvoice_dialog.py, with distillation support in zipvoice/models/zipvoice_distill.py. The zipvoice/luxvoice.py file exposes the LuxTTS class that serves as the primary user-facing interface, coordinating between tokenizer, feature extraction, and the underlying model.

Inference flow originates from LuxTTS.encode_prompt() in luxvoice.py, which processes a reference audio file through feature extraction (zipvoice/utils/feature.py and zipvoice/bin/compute_fbank.py) and tokenization (zipvoice/tokenizer/tokenizer.py, zipvoice/tokenizer/normalizer.py). The encoded prompt then feeds into LuxTTS.generate_speech(), which coordinates the diffusion sampling process. The zipvoice/bin/infer_zipvoice.py script mirrors this pipeline for command-line usage, while zipvoice/bin/infer_zipvoice_onnx.py and zipvoice/bin/tensorrt_export.py provide export paths for accelerated inference.

Outside the core package, pyproject.toml and requirements.txt declare dependencies (Transformers, torch, soundfile, etc.), and the README documents Colab and Hugging Face Space integration points.

How It Is Wired

Execution enters through LuxTTS in zipvoice/luxvoice.py. The __init__ method loads model weights (Hugging Face hub path or local checkpoint) and device placement (CUDA, CPU, or MPS). Two primary public methods orchestrate the pipeline:

  • encode_prompt(audio_path, duration, rms) — loads a reference audio file, computes fbank features via zipvoice/utils/feature.py (invoking zipvoice/bin/compute_fbank.py logic), normalizes through the tokenizer, and returns a condensed encoding. This is the heaviest init-time step (∼10s first call due to librosa).
  • generate_speech(text, encoded_prompt, num_steps, t_shift, speed, return_smooth, ...) — takes the text prompt and encoded reference, runs the zipvoice diffusion model (zipvoice/models/zipvoice.py) for num_steps sampling iterations, and returns the generated waveform at 48khz.

The call graph funnels through luxvoice.py as a hub: both encode_prompt and generate_speech are the only external entry points. Downstream, encode_prompt touches the tokenizer, feature extractor, and potentially the Hugging Face hub for model weights. generate_speech depends on the model architecture file and passes through the scheduler (zipvoice/utils/lr_scheduler.py) and optimizer utilities (zipvoice/utils/optim.py) if training-mode hooks are active. The bin scripts (infer_zipvoice.py, infer_zipvoice_dialog.py, etc.) are thin wrappers that instantiate LuxTTS and invoke these same methods with CLI-parsed arguments.

No export cycle leaves the repository's control flow without explicit user invocation—ONNX and TensorRT exports are triggered via zipvoice/bin/onnx_export.py and zipvoice/bin/tensorrt_export.py respectively, but those are separate workflows, not part of the default inference path.

How To Use It

Setup

git clone https://github.com/moses-y/LuxTTS.git
cd LuxTTS
pip install -r requirements.txt

The pyproject.toml lists runtime dependencies; no lockfile is present, so reproducible builds require pinning versions manually or using a venv.

Configuration

No environment variables or secret keys are required for basic operation. The LuxTTS constructor accepts a device argument ('cuda', 'cpu', 'mps') and optional threads for CPU mode. Model weights are loaded from the Hugging Face hub by default (e.g., 'YatharthS/LuxTTS') or from a local path if specified.

Running it

from zipvoice.luxvoice import LuxTTS

lux_tts = LuxTTS('YatharthS/LuxTTS', device='cuda')

prompt_audio = 'reference.wav'
encoded_prompt = lux_tts.encode_prompt(prompt_audio, rms=0.01)

final_wav = lux_tts.generate_speech("Hello world", encoded_prompt, num_steps=4)

import soundfile as sf
final_wav = final_wav.numpy().squeeze()
sf.write('output.wav', final_wav, 48000)

The entry point script is zipvoice/bin/infer_zipvoice.py for CLI-based generation without writing a Python script.

Real-World Use

A practical workflow: a developer wants to clone a user's voice for a voice assistant without provisioning a multi-GPU cluster. They clone the repo, install requirements, and run the Python snippet above with a 3+ second WAV reference. The model outputs 48khz audio in under a second on a single RTX 3090, and realtime on CPU with threads=2. The low VRAM footprint (≤1GB) means it runs on laptops with discrete GPUs or even high-end Macs via MPS. The 48khz sample rate makes it suitable for fidelity-critical applications like audiobook narration or dubbing, where 24khz models introduce noticeable high-frequency muffling.

Code Health & Issues

  • [Medium/SDLC] No test files detected — repository-wide, no automated test coverage for model paths, encoding, or generation pipelines. Verification relies on manual invocation.
  • [Medium/SDLC] No CI/CD pipeline detected — no .github/workflows or equivalent build/test gate. Commits are not validated for importability or regression.
  • [Low/Risk] Dependencies declared without a lockfile — pyproject.toml lists requirements but no requirements-lock.txt or similar is present. Upgrades to transformer/torch versions may change model behavior or break imports without explicit pinning.

Beyond these heuristic findings, the repository structure is coherent: the zipvoice/ package owns all model, tokenizer, and utility logic, while the root contains only packaging and documentation files. No source code outside zipvoice/ touches the model's internal data flow.

The Bottom Line

This is a focused, efficiently engineered TTS implementation that delivers on its speed and VRAM claims. The LuxTTS class in zipvoice/luxvoice.py provides a clean, minimal API; the internal wiring through zipvoice/models/ and zipvoice/utils/ is straightforward enough to modify or extend. The absence of tests and CI means any changes to the modeling pipeline require manual verification, and dependency upgrades should be treated as breaking until confirmed. Use case: individual developers or small teams needing high-fidelity voice cloning on a single GPU without the overhead of larger frameworks (e.g., FastSpeech2, VITS, or Tortoise). Large organizations requiring guaranteed reproducibility or automated regression will need to add their own test harness and lockfile discipline.