The Problem

Running modern speech models (TTS, STT, audio codecs) on Apple Silicon usually means bridging Python MLX code into a Swift app or shelling out to a server. That adds latency, packaging pain, and a second runtime. This repo is a native Swift SDK that runs those models directly on-device via MLX, so a macOS/iOS app can do text-to-speech, speech-to-text, and audio tokenization without leaving the process.

What This Does

mlx-audio-swift is a modular Swift package. Sources/MLXAudioCore holds base types, DSP helpers (DSP.swift), and audio session management (AudioSessionManager.swift). Sources/MLXAudioCodecs implements codecs — SNAC, Vocos, Mimi, Encodec, DACVAE — that convert audio to/from tokens. Sources/MLXAudioTTS implements TTS models (Soprano, Qwen3, LlamaTTS, PocketTTS, Marvis), and Sources/MLXAudioSTT implements ASR (GLMASR, Qwen3ASR). Examples/VoicesApp is a working SwiftUI app exercising TTS with voice selection.

Models load from HuggingFace Hub by name, e.g. SopranoModel.fromPretrained("mlx-community/Soprano-80M-bf16"). Generation is async and supports streaming via generateStream, yielding .token, .audio, and .info events.

How It Is Wired

The static import graph for this repo resolved 0 internal modules and 0 edges across the 77 Swift files, so the module-level wiring has not been mapped by this analysis. The README documents the entry points: SopranoModel.fromPretrained (TTS) and GLMASRModel.fromPretrained (STT) in Sources/MLXAudioTTS/Models/Soprano/Soprano.swift and Sources/MLXAudioSTT/Models/GLMASR/GLMASR.swift. From there, generate(...) and generateStream(...) produce audio arrays saved via saveAudioArray(...) in MLXAudioCore. The codecs expose encode/decode (e.g. SNAC.fromPretrained → snac.encode(audio)).

Each effect is owned by a module: model downloads go through ModelUtils.swift in MLXAudioCore; audio I/O through AudioUtils.swift and AudioPlayerManager.swift; session behavior through AudioSessionManager.swift. Without a resolved call graph, I can't name the exact hub functions or predict blast radius per file — that wiring is unverified.

How To Use It

Add the package via Swift Package Manager:

dependencies: [
    .package(url: "https://github.com/Blaizzy/mlx-audio-swift.git", branch: "main")
]

Import only what you need: MLXAudioTTS, MLXAudioCore, etc. No environment variables are required — models download automatically from HuggingFace. The Examples/VoicesApp needs its .xcconfig files; Local.xcconfig.template suggests copying it to Local.xcconfig for local keys. Requirements are macOS 14+ / iOS 17+ on Apple Silicon. There is no CLI entry point; usage is programmatic via the model APIs.

Real-World Use

A macOS dictation utility that transcribes microphone audio with GLMASRModel, then reads the result back with SopranoModel.generate(...), streaming tokens to a status label while audio plays through AudioPlayerManager. Both models run locally with no network round-trip after the initial HuggingFace download.

Code Health & Issues

Static analysis (not opinion) found 42 issues: 9 high, 33 medium. High-severity findings are deep nesting (24 instances, max depth 8, e.g. Examples/VoicesApp/VoicesApp/ViewModels/TTSViewModel.swift) and duplicated code (302 repeated 6-line blocks across 25 files, concentrated in Sources/MLXAudioCodecs/.../Layers.swift files). Medium findings: 5 oversized files (e.g. Sources/MLXAudioTTS/Models/Qwen3/Qwen3.swift at 1007 lines) and 12 high-branching-density files (25 branch points over 74 lines in DACVAEConfig.swift). Fixes: guard clauses/early returns, shared helpers, splitting oversized files.

The CI audit adds three items: High — pin GitHub Actions to commit SHAs (.github/workflows uses maxim-lobanov/setup-xcode@v1); Medium — declare least-privilege permissions: contents: read in .github/workflows/tests.yaml; Low — add timeout-minutes to jobs. Tests and an MIT license exist; no lockfile is committed.

The Bottom Line

Solid, genuinely modular Swift SDK for on-device speech on Apple Silicon, with a real example app and CI. The duplicated codec layers and oversized model files will make adding a new model tedious, and the CI token posture needs hardening before this runs in shared repos. Best for Swift teams that want native, offline TTS/STT without a Python sidecar.