VibeVoice

The Problem

Clients that need on‑premise speech‑to‑text (ASR) or streaming text‑to‑speech (TTS) often have to stitch together separate models, data‑pre‑processing utilities, and demo servers. The result is duplicated code, unclear runtime paths, and fragile builds that change with each dependency update.

What This Does

VibeVoice is a lightweight portfolio of four related demos and libraries:

vibevoice/ – core model wrappers, tokenizers and streaming inference code (modeling_vibevoice*.py, processor/).
vllm_plugin/ – a thin adapter that lets the ASR models run under the vLLM inference server (model.py, scripts/gradio_asr_demo_api_video.py).
demo/ – Flask web UI (demo/web/app.py), Gradio demos, and a CLI for realtime inference (realtime_model_inference_from_file.py).
finetuning-asr/ – scripts and README for fine‑tuning the ASR model on custom data.

All code is Python; the only build artifact is pyproject.toml. The repository ships pre‑trained model checkpoints in demo/voices/streaming_model/ and a handful of media files for the demos.

How It Is Wired

Execution starts at the concrete entry points:

Entry point	File	Primary flow
Flask web server	`demo/web/app.py` → `load()` → `model.generate`	`load()` (line 70) builds a `VibeVoice` instance, then `_run_generation` (line 202) calls `model.generate`. The generate path traverses `modeling_vibevoice_streaming_inference.__init__`, `_update_model_kwargs_for_generation`, and finally the underlying Hugging‑Face `generate` routine.
Realtime CLI	`demo/realtime_model_inference_from_file.py::main`	`main()` parses args, calls `setup_voice_presets`, then invokes `model.generate` on audio chunks. The same `modeling_vibevoice_streaming_inference` code is hit, meaning any change to that module ripples through both the web UI and the CLI.
vLLM ASR endpoint	`vllm_plugin/model.py::load_file` → `model.generate`	The plugin loads a checkpoint via `load_file`, wraps it in the same `VibeVoice` class, and forwards inbound HTTP requests to `model.generate`.

The internal call graph shows 323 intra‑repo edges. The most used symbols (set, get_input_embeddings, from_pretrained, AudioNormalizer, load_audio_use_ffmpeg) are imported by 4–9 callers each, giving them the widest blast radius. No circular imports were detected, but several modules have deep nesting (up to 10 levels) and large line counts (e.g., modular_vibevoice_tokenizer.py with 917 lines), which makes reasoning about changes harder.

External effects are limited to:

Model inference (self.model.generate) – no direct DB access.
Filesystem writes (e.g., save_audio in the CLI).
External commands via ffmpeg wrappers (run_command).
Outbound network calls for the vLLM demo (HTTP to the inference server).

Thus the repo’s side‑effects are confined to I/O and model inference, and the control‑flow hot‑spots are clearly identified in the entry‑point table above.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/VibeVoice
cd VibeVoice

# Install the package in editable mode (pyproject.toml defines the deps)
pip install -e .

# Run the Flask demo (requires a GPU for model loading)
python demo/web/app.py   # starts a local server at http://127.0.0.1:5000

# Or run the realtime CLI on a local audio file
python demo/realtime_model_inference_from_file.py --input path/to/audio.wav

The README and docs/ folder contain additional usage notes (e.g., Gradio demo commands). No environment‑variable configuration is required out of the box; the scripts locate model checkpoints relative to the demo/voices/streaming_model/ directory.

Real‑World Use

A product team can embed the vibevoice library in a microservice that receives audio blobs via HTTP, calls model.generate for transcription, and returns a JSON payload with speaker‑attributed timestamps. The Flask demo already shows this pattern, and the vLLM plugin demonstrates how to expose the same inference through a high‑throughput server.

Code Health & Issues

High – No lockfile – pyproject.toml lists version ranges but no poetry.lock/uv.lock. Unlocked dependencies risk divergent builds. Fix: generate and commit a lockfile with the chosen package manager.
High – No CI – 37 source files, 0 CI configuration. Changes are not automatically built or tested. Fix: add a GitHub Actions workflow that runs pip install -e . && pytest.
Medium – No Dependabot – Only one manifest, no automated vulnerability updates. Fix: add .github/dependabot.yml covering python and github-actions.
Medium – Large binaries in Git – Media files (demo2-song.mp4 23 MB, several model checkpoints ≈ 6 MB each) inflate clone size. Fix: migrate these assets to Git LFS or external storage and fetch them in a setup script.

Additional observations: the repo includes a license (LICENSE), a CONTRIBUTING.md, and two test files, but lacks Docker support and any secret scanning failures.

The Bottom Line

VibeVoice provides ready‑to‑run ASR/TTS demos and a reusable Python library that can be integrated into custom services. The codebase is functional but suffers from maintainability problems—deeply nested, oversized modules and missing build safeguards. Engineers comfortable with PyTorch and Flask can extract the core inference path, but should first add a lockfile, CI, and consider refactoring the heaviest modules to reduce cognitive load.