The Problem
Transcribing speech on‑device or in low‑resource environments often requires heavy Python stacks (PyTorch, vLLM, CUDA) and large runtime dependencies. Teams that need a lightweight, self‑contained inference engine for the Voxtral Realtime 4B model lack a reference implementation that runs directly from C without a Python interpreter or GPU‑specific toolkits.
What This Does
The repository supplies a pure‑C inference pipeline for the Mistral Voxtral Realtime 4B speech‑to‑text model. Core components live in the voxtral.c/.h files:
voxtralencoder.c / voxtraldecoder.c – implement the chunked audio encoder and transformer decoder. voxtralkernels.c + voxtralmetal.m – provide CPU (BLAS) and Metal‑GPU kernels for attention and matrix ops. voxtraltokenizer.c – converts model logits to token strings. voxtralsafetensors.c – memory‑maps the BF16 weights from the Safetensors file, avoiding full loading.
The command‑line driver main.c (built as ./voxtral) exposes a simple CLI (-d <modeldir> -i <wav> or --stdin) and a streaming C API (voxstreamt) that can be linked into other applications. The pythonsimpleimplementation.py mirrors the same flow in Python for reference, but the C binary is fully independent of any Python runtime.
How To Use It
Build (choose backend) make mps # Apple Silicon – Metal acceleration (fastest) or make blas # Intel/macOS/Linux – OpenBLAS fallback Pull the model (≈8.9 GB) ./downloadmodel.sh # places model files under ./voxtral-model Transcribe a WAV file ./voxtral -d voxtral-model -i samples/testspeech.wav Stream any audio format via ffmpeg ffmpeg -i audio.mp3 -f s16le -ar 16000 -ac 1 - 2>/dev/null | \ ./voxtral -d voxtral-model --stdin
No external libraries are required for the binary itself; the only optional dependency is OpenBLAS when the blas target is used. All configuration is passed via CLI flags; there is no separate config file.
Real‑World Use
A voice‑assistant service running on a macOS edge device can embed voxtral.c as a static library, instantiate voxstreamt, feed microphone PCM buffers, and emit token strings to downstream NLU components in real time. Because the KV cache is circular and capped at 8192 positions, memory usage stays bounded even for hour‑long recordings.
voxstreamt st = voxstreamnew("voxtral-model"); voxstreamfeed(st, pcmchunk, chunklen); char *text = voxstreamnexttoken(st); printf("%s ", text);
Code Health & Issues
Medium – No CI/CD – No .github/workflows or other automation; build/test gating is manual. Low – Limited test coverage – Only two test files (inspectweights.c, voxtraldecoder.c‑related checks) exist; core audio pipeline and streaming API lack unit tests. Medium – Platform‑specific kernels – Metal implementation (voxtralmetal.m, voxtralshaders.metal) works only on Apple Silicon; fallback to BLAS may be slow due to BF16→FP32 conversion noted in README. Low – Sparse documentation – README covers build and run steps, but internal APIs (voxstream_t, weight loading) have no inline docs or examples beyond the CLI. Low – License present – LICENSE file exists; no apparent proprietary code. Low – No runtime error handling – Functions like fopen or mmap are used without exhaustive checks, which could cause crashes on missing model files.
Overall the code is compact and self‑contained, but the lack of automated testing and limited platform support increase integration risk.
The Bottom Line
voxtral.c delivers a functional, zero‑dependency C inference engine for the Voxtral Realtime 4B model, suitable for low‑overhead or Apple‑silicon deployments. It is a solid reference implementation, but production use should account for the modest test suite, missing CI, and the performance penalty of BF16‑to‑FP32 conversion on non‑Metal backends. Teams comfortable with C integration and willing to add their own validation harness will find it valuable.