speech to speech - Moses Yebei

The Problem

Building local voice agents can be a pain in the neck, especially when you're juggling multiple models for voice activity detection (VAD), speech-to-text (STT), language processing, and text-to-speech (TTS). You’re stuck either integrating a bunch of libraries with varying quality or relying on some half-baked cloud service that eats into your budget and time.

What This Does

Enter the speech-to-speech repo. It gives you an open-source pipeline that covers the full deal: VAD, STT, language modeling, and TTS, all wrapped in a neat package. The core structure is split into directories: VAD for voice detection, STT for transcribing audio, LLM for language modeling, and TTS for generating speech.

You’ll find files like STT/whisperstthandler.py to handle Whisper model integration and TTS/chatTTShandler.py for generating speech responses. The s2spipeline.py file ties everything together, allowing you to run the complete pipeline in a few lines of code. It’s modular enough that you can swap models in and out without losing your mind.

Real-World Use

Imagine you want to set up a local assistant that converts speech to text, processes it, and reads it back in a different voice. You can clone the repo, set up your environment with a couple of uv commands, and run something like this:

With minimal fuss, you get a responsive voice agent that works offline. If you’re on Mac and using Melo TTS, just remember to download the necessary files and handle any compatibility issues.

The Bottom Line

This repo is solid if you're diving into voice agents without wanting to wrestle with a dozen different libraries. It's not for tiny projects—if you need a simple TTS, you're better off with a standalone solution. But for those needing a full-featured local agent, this is a good start. Just make sure your system can handle the requirements—especially if you're on Apple Silicon.

What the analyser found

Deployment readiness

3/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

70 files

Python56
Markdown4
YAML1

Docker

Architecture

Top-level areas of the codebase, sized by module count. Arrows show how many imports cross from one area into another.

graph TD g0["arguments_classes 20 modules"] g1["STT 8 modules"] g2["TTS 8 modules"] g3["(root) 5 modules"] g4["LLM 5 modules"] g5["connections 4 modules"] g6["utils 3 modules"] g3 -->|20| g0 g3 -->|13| g1 g3 -->|12| g2 g1 -->|7| g3 g2 -->|7| g3 g4 -->|4| g3 g3 -->|4| g5 g3 -->|4| g4 g2 -->|2| g6 g4 -->|1| g6 g1 -->|1| g6 g3 -->|1| g6

Module dependencies

The 10 most depended-upon modules of 56, from static import analysis. Red outline marks a module in an import cycle.

graph LR n0_baseHandler["baseHandler"] n1_STT_parakeet_tdt_handler["parakeet_tdt_handler"] n2_LLM_chat["chat"] n3_utils_mlx_lock["mlx_lock"] n4_TTS_kokoro_handler["kokoro_handler"] n5_STT_faster_whisper_handler["faster_whisper_handler"] n6_ightning_whisper_mlx_handler["lightning_whisper_mlx_ha"] n7_TT_mlx_audio_whisper_handler["mlx_audio_whisper_handle"] n8_STT_moonshine_handler["moonshine_handler"] n9_STT_whisper_stt_handler["whisper_stt_handler"] n5_STT_faster_whisper_handler --> n0_baseHandler n6_ightning_whisper_mlx_handler --> n0_baseHandler n7_TT_mlx_audio_whisper_handler --> n0_baseHandler n8_STT_moonshine_handler --> n0_baseHandler n1_STT_parakeet_tdt_handler --> n0_baseHandler n1_STT_parakeet_tdt_handler --> n3_utils_mlx_lock n9_STT_whisper_stt_handler --> n0_baseHandler n4_TTS_kokoro_handler --> n0_baseHandler n4_TTS_kokoro_handler --> n3_utils_mlx_lock