speech to speech

The Problem

Deploying a local voice‑assistant that runs end‑to‑end speech‑to‑speech (VAD → STT → LLM → TTS) typically requires stitching together several heavyweight libraries, handling device‑specific builds, and writing glue code that is hard to reuse. Teams that want an on‑prem solution must also manage Docker images, model downloads, and a fragile command‑line interface.

What This Does

The speech‑to‑speech repository provides a ready‑made, modular pipeline that ties open‑source models together:

• VAD – VAD/vad_handler.py and VAD/vad_iterator.py wrap Silero VAD.
• STT – nine handlers under STT/ (e.g., parakeet_tdt_handler.py, whisper_stt_handler.py) expose setup, warmup, and process methods for a variety of Whisper‑based back‑ends.
• LLM – LLM/language_model.py (and its concrete subclasses) implement setup, warmup, and process for Hugging‑Face transformers, mlx‑lm, or OpenAI API.
• TTS – ten handlers under TTS/ (e.g., kokoro_handler.py, facebookmms_handler.py) follow the same three‑method pattern.

Argument definitions live in arguments_classes/ so each handler can be launched from the CLI or embedded in another service without code changes. The top‑level script baseHandler.py wires the four stages together and offers a run entry point that can be called programmatically.

How It Is Wired

Execution begins at benchmark_stt.py:main (line 330) for the STT benchmark and at baseHandler.py:run for the full pipeline.

• main → setup (calls warmup × 15) → process → model inference via self.client.chat.completions.create.
• run → s2s_pipeline.setup_logger → s2s_pipeline.parse_arguments → s2s_pipeline.rename_args → s2s_pipeline.check_mac_settings.

The central hub is baseHandler.py, which is imported by 19 other modules (highest fan‑in). It creates the VAD, STT, LLM, and TTS handler objects and sequences their setup/warmup/process calls.

Key downstream modules:

The import graph shows 56 modules with 85 import edges and no circular dependencies, indicating a clean top‑level structure. However, the deepest nesting (6 levels) appears in baseHandler.py, STT/parakeet_tdt_handler.py, and connections/websocket_streamer.py, making those sections harder to follow.

How To Use It

# Clone the fork that contains the local modifications
git clone https://github.com/moses-y/speech-to-speech.git
cd speech-to-speech

# Build the Docker image (includes PyTorch, CUDA, and all handlers)
docker build -t s2s:latest -f Dockerfile .

# Run the full pipeline inside the container
docker run --gpus all -v $(pwd)/models:/models -e HF_HOME=/models \
    s2s:latest python baseHandler.py --config arguments_classes/module_arguments.py

The repository supplies a requirements.txt for local Python execution; install with pip install -r requirements.txt if you prefer a non‑Docker run.

Configuration arguments are defined in the arguments_classes/ files (e.g., language_model_arguments.py). No secret keys are hard‑coded; OpenAI API usage requires setting OPENAI_API_KEY in the environment before invoking the LLM handler.

Real‑World Use

A company building an on‑prem customer‑service kiosk can launch the container, mount a local model cache, and call baseHandler.run() from a Flask endpoint. The endpoint streams microphone input to VAD/vad_handler.py, feeds transcriptions to STT/parakeet_tdt_handler.py, sends the text to LLM/openai_api_language_model.py, and finally streams the generated audio from TTS/kokoro_handler.py back to the client.

Code Health & Issues

• HIGH – No CI pipeline. 56 source files lack any GitHub Actions or other automated build/test configuration.
• HIGH – No Docker build gate. The Dockerfile is never validated by an automated workflow.
• MEDIUM – Dependabot/Renovate missing. requirements.txt is not locked, leaving builds non‑reproducible.
• MEDIUM – Base image not pinned by digest (pytorch/pytorch:2.4.0-cuda12.1-cudnn9-devel).
• MEDIUM – Test coverage is minimal (1 test file vs. 56 source files).

Additional observations from the static analysis:

• Deep nesting (up to 6 levels) in three files makes future modifications risky.
• baseHandler.py is a high‑impact hub; any change may affect 19 dependent modules.
• Repeated 6‑line code blocks appear across 14 files in the LLM and STT packages – a DRY refactor would reduce maintenance load.
• Broad except: clauses in VAD/vad_iterator.py, connections/local_audio_streamer.py, and connections/websocket_streamer.py swallow errors, hindering debugging.

The Bottom Line

The repo delivers a functional, Docker‑ready speech‑to‑speech stack with clear separation of VAD, STT, LLM, and TTS concerns, making it a solid starting point for on‑prem voice agents. Production readiness is limited by missing CI/CD, sparse testing, and some code‑quality hotspots; addressing those will be essential before extensive customization or enterprise deployment.