The Problem

Developers targeting Apple‑silicon need a speech stack that runs entirely on‑device, avoids the latency of cloud APIs, and can be embedded in Python or a web UI. Existing toolkits either rely on heavyweight frameworks (PyTorch, TensorFlow) or lack native Apple‑MLX acceleration, making real‑time TTS/STT/STS on M‑series chips impractical.

What This Does

mlx-audio implements end‑to‑end speech models (TTS, STT, STS) on top of Apple’s MLX tensor library. The core Python package lives in mlx_audio/ (≈540 files) and includes:

  • Model loaders and inference in mlx_audio/tts/, mlx_audio/stt/, mlx_audio/sts/.
  • Low‑level audio I/O (mlx_audio/audio_io.py), DSP utilities (mlx_audio/dsp.py), and a generic codec layer (mlx_audio/codec/).
  • A small HTTP server (mlx_audio/server.py) exposing a REST API and a browser UI in mlx_audio/ui/ (React/Next.js).

Key entry points are:

  • CLI – mlx_audio/tts/utils.py provides the mlx_audio.tts.generate console command.
  • Server – mlx_audio/server.py:main (line 893) starts the API and UI.
  • Programmatic API – mlx_audio/tts/utils.load_model returns a model object whose generate method yields Result objects containing an mx.array waveform.

How It Is Wired

Execution begins at mlx_audio/server.py:main. The function builds a FastAPI app, registers routes, and then calls add_model (line 227) to load a default model. add_model invokes load_model → from_pretrained, which reads model files from the filesystem (e.g., shutil.copy of a cached checkpoint).

When a client POSTs to /tts, the handler calls mlx_audio/tts/models/base.generate. Inside generate the call chain includes:

  • eval (called 173 times across the repo) to switch model mode,
  • clear_cache (36 calls) to free MLX buffers,
  • pad (35 calls) for waveform alignment,
  • norm (65 calls) for audio post‑processing.

The blast‑radius modules are:

  • mlx_audio/utils.py – 38 inbound imports, 6 outbound; part of a circular import with mlx_audio/tts/utils.py and mlx_audio/stt/utils.py.
  • mlx_audio/tts/models/base.py – 40 inbound imports, no outbound imports; a hub for all TTS model classes.

The import graph contains 25 cycles; the most visible cycle is among the three utils files, which increases maintenance risk because a change in one forces reloads of the others.

Large, deeply nested files (mlx_audio/tts/models/qwen3_tts/qwen3_tts.py, mlx_audio/stt/models/wav2vec/wav2vec.py, mlx_audio/tts/models/bailingmm/bailingmm.py) each exceed 2 k lines and have indentation depths of 8, making them hard to reason about and error‑prone.

External interactions are limited to:

  • Filesystem reads/writes (31 functions, e.g., model checkpoint loading).
  • A handful of network calls (9 functions) performed by the REST server.
  • One subprocess call to ffmpeg for audio conversion (mlx_audio/convert.py).

No database or persistent secret handling is present.

How To Use It

# Install the core library (Python 3.11+, Apple Silicon)
pip install mlx-audio

# Run the built‑in server (exposes /tts, /stt, /sts)
python -m mlx_audio.server
# Server starts on http://127.0.0.1:8000 and serves the React UI.

# One‑off TTS from the CLI
mlx_audio.tts.generate \
    --model mlx-community/Kokoro-82M-bf16 \
    --text "Hello, world!" \
    --lang_code a \
    --output_path ./out

The CLI documentation lives in README.md; the server UI source is under mlx_audio/ui/. No additional configuration files are required beyond the optional .env for a custom HuggingFace token (refer to mlx_audio/server.py where os.getenv("HF_TOKEN") is read).

Real‑World Use

A media‑playback app on macOS can embed mlx-audio to generate voice‑overs on‑device:

from mlx_audio.tts.utils import load_model

model = load_model("mlx-community/Kokoro-82M-bf16")
for chunk in model.generate("Chapter 1 intro", voice="af_heart"):
    play(chunk.audio)          # mx.array → numpy → sounddevice.play

All inference runs on the M‑series GPU/CPU via MLX, delivering sub‑second latency without external bandwidth.

Code Health & Issues

  • High – Pin GitHub Actions – .github/workflows/* uses pypa/gh-action-pypi-publish@v1.13.0. Replace @v1.13.0 with a full SHA to avoid supply‑chain drift.
  • Medium – Least‑privilege GITHUB_TOKEN – tests.yml declares no permissions; add permissions: contents: read.
  • Medium – Enable Dependabot – No dependabot.yml present; add one for Python, npm, and GitHub Actions.
  • Medium – Dependency‑vulnerability scan – CI lacks a review step; integrate dependency-review-action or osv-scanner.
  • Medium – Checkout persist‑credentials – python-publish.yml should set persist-credentials: false.
  • Low – Job timeouts – No timeout-minutes in CI jobs; add a reasonable limit (e.g., 30 min).
  • Low – Missing lockfile – examples/bible-audiobook/package.json has no package-lock.json; consider committing a lockfile for reproducible npm builds.

The repo includes a full test suite (mlx_audio/*/tests/) and CI runs on every PR, but the import cycles and oversized modules increase the risk of regression when extending functionality.

The Bottom Line

mlx-audio delivers a functional, Apple‑MLX‑accelerated speech stack with a usable CLI, Python API, and web UI, making on‑device TTS/STT/STS feasible on M‑series hardware. However, the codebase suffers from import cycles, very large files, and deep nesting, which raise the maintenance burden. It is suitable for teams comfortable navigating a complex Python project and willing to address the identified health issues.