The Problem
Developers building audio‑centric apps on Apple Silicon need a Swift‑native way to run ML‑based speech synthesis, recognition, and codec processing without pulling in heavyweight Python runtimes. Existing solutions either require cross‑language bridges or are limited to a single model family, increasing app size and integration effort.
What This Does
mlx-audio-swift delivers a modular Swift SDK that wraps the MLX framework for on‑device neural audio. The source tree is split into clear modules:
MLXAudioCore (Sources/MLXAudioCore/.swift) supplies low‑level utilities—audio loading (AudioUtils.swift), session handling (AudioSessionManager.swift), and DSP helpers (DSP.swift). MLXAudioCodecs (Sources/MLXAudioCodecs/) implements codec pipelines such as SNAC (SNAC/SNACDecoder.swift) and Vocos (Vocos/Vocos.swift). MLXAudioTTS (Sources/MLXAudioTTS/) provides TTS models (Soprano, Qwen3, Llama) with async‑await generation APIs (Generation.swift, model‑specific generate methods). MLXAudioSTT (Sources/MLXAudioSTT/) offers speech‑to‑text models (GLMASR, Whisper) exposing a simple generate(audio:) call. MLXAudioUI (Sources/MLXAudioUI/Placeholder.swift) contains SwiftUI components for quick UI prototyping, demonstrated in the Examples/VoicesApp Xcode project.
The SDK’s design lets a consumer import only the needed product, e.g. .product(name: "MLXAudioTTS", package: "mlx-audio-swift"), keeping binary size low.
How To Use It
Setup
Add the package via Swift Package Manager (SPM) as shown in README.md:
// Package.swift (root) dependencies: [ .package(url: "https://github.com/Blaizzy/mlx-audio-swift.git", branch: "main") ], targets: [ .target( name: "MyApp", dependencies: [ .product(name: "MLXAudioTTS", package: "mlx-audio-swift"), .product(name: "MLXAudioCore", package: "mlx-audio-swift") ] ) ]
Open the Xcode workspace Examples/VoicesApp/VoicesApp.xcodeproj to explore a runnable demo.
Configuration
No explicit environment variables are required. Model assets are fetched automatically by the SDK’s fromPretrained static constructors (e.g. SopranoModel.fromPretrained). The only configuration point is the optional .xcconfig files under Examples/VoicesApp/Config/ that control build settings.
Running
A minimal TTS flow can be placed in any Swift file:
import MLXAudioTTS import MLXAudioCore
let model = try await SopranoModel.fromPretrained("mlx-community/Soprano-80M-bf16") let audio = try await model.generate( text: "Hello from MLX Audio Swift!", parameters: GenerateParameters(maxTokens: 200, temperature: 0.7, topP: 0.95) ) try saveAudioArray(audio, sampleRate: Double(model.sampleRate), to: URL(fileURLWithPath: "/tmp/out.wav"))
STT usage mirrors the pattern shown in README.md and is implemented in Sources/MLXAudioSTT/MLXAudioSTT.swift.
Real‑World Use
A podcast‑editing app can embed the SDK to transcribe recorded segments on‑device and then synthesize voice‑overs without network latency:
let stt = try await GLMASRModel.fromPretrained("mlx-community/GLM-ASR-Nano-2512-4bit") let transcript = stt.generate(audio: rawAudio).text
let tts = try await Qwen3Model.fromPretrained("mlx-community/VyvoTTS-EN-Beta-4bit") let voiceover = try await tts.generate(text: transcript, parameters: .default) audioPlayer.play(voiceover)
All processing stays local, preserving user privacy and enabling offline operation.
Code Health & Issues
Bugs / Risks – Medium – AudioPlayerManager.swift lacks explicit error propagation when AVAudioEngine fails to start; callers must check nil returns. Thread Safety – Low – AudioSessionManager manipulates AVAudioSession on the main thread only; background calls could cause race conditions. Testing Coverage – Medium – Six test files (Tests/.swift) cover codec and model loading but do not exercise streaming APIs or UI components. CI – Low – GitHub Actions workflow (.github/workflows/tests.yaml) runs swift test on macOS, providing basic verification. Documentation – Low – README covers core usage; individual modules lack doc comments, making IDE discoverability limited. License – None indicated in source files; repository includes a top‑level LICENSE (MIT) which is sufficient. Dependency Hygiene – Low – No explicit version pinning for the MLX framework; updates rely on the upstream package manager, which could introduce breaking changes.
Overall the repo follows Swift conventions, separates concerns cleanly, and ships a functional CI pipeline.
The Bottom Line
mlx-audio-swift offers a well‑structured, Swift‑first interface to MLX audio models, ideal for Apple‑silicon apps that need on‑device speech capabilities while keeping binary size under control. The SDK is usable out‑of‑the‑box, though developers should add error handling around audio session startup and consider extending test coverage for streaming scenarios. Suitable for teams building privacy‑focused media or voice‑assistant features on macOS/iOS.