The Problem

KV cache memory is the primary bottleneck for running large language models locally. A 35B-parameter MoE model with a 32K context can consume tens of gigabytes of KV cache alone, forcing users to choose between context length and model size. Existing 4-bit quantization methods (q40) deliver 4x compression but degrade quality noticeably.

What This Does

TurboQuant+ implements the TurboQuant KV cache compression method (ICLR 2026) from Google Research, combining PolarQuant with Walsh-Hadamard rotation to achieve 4.6x compression with quality loss of only 1% versus q80. The core implementation lives in turboquant/ — polarquant.py handles the polar decomposition, rotation.py implements the Walsh-Hadamard transform, and codebook.py manages the quantized codebooks. The outlier.py module extends the base paper with outlier-aware bit allocation.

The project includes a working C port integrated into llama.cpp with Metal GPU kernels. Benchmarks on Apple Silicon (M5 Max) show prefill speed at 99% of q80 across 2K-32K context lengths, with decode speed at 88-92% for typical contexts. The benchmarks/ directory contains the full comparison suite, and docs/ documents the optimization journey and quality results in detail.

How To Use It

The repo is a Python package with a pyproject.toml — install with pip install -e . or uv sync. There are no published releases or Docker images.

pip install -e . python benchmarks/runbenchmark.py

The benchmarks/ directory contains the entry points: runbenchmark.py for the main comparison suite, demo.py for a quick demonstration, and shell scripts (benchmarkllama.sh) for running against real llama.cpp builds. The turboquant/ package exports the quantization functions directly for programmatic use. The profiles/baseline-m5-max-128gb.json file holds hardware-specific baseline data.

Missing from the repo: no documented CLI entry point, no example script showing how to use the Python API, and no instructions for building the llama.cpp integration. The README references the C port but doesn't explain how to reproduce it.

Real-World Use

For a developer running llama.cpp on Apple Silicon with large context windows:

from turboquant import turboquant from turboquant.rotation import rotatetensor

Compress a KV cache tensor

rotated = rotatetensor(kvtensor, dim=-1) compressed = turboquant(rotated, bits=3)

The llama.cpp integration works via --cache-type-k turbo3 --cache-type-v turbo3, enabling 48K context on a 35B model with 128GB RAM. The NIAH retrieval tests show 80% accuracy at 32K context versus 85% for q80 — acceptable for most retrieval workloads.

Code Health & Issues

Med — No lockfile: pyproject.toml declares dependencies without pinned versions, so builds aren't reproducible. This is a real concern for a project claiming production readiness. Med — Single-platform focus: Everything is benchmarked on Apple Silicon/Metal. No evidence of CPU or CUDA support, limiting the addressable audience. Low — Documentation gap: 20 doc files exist, but the README doesn't explain how to build the llama.cpp integration or reproduce the C port. The scripts/ directory has some tooling but no usage docs. Low — Test coverage is strong (511 tests, 100% on diagnostics) and CI exists via GitHub Actions, but the test suite is Python-only — the C/Metal code has no automated tests in this repo.

The Bottom Line

TurboQuant+ delivers a real, working 4.6x KV cache compression with quality and speed that hold up against q80 — a genuinely useful result for local LLM inference on Apple Silicon. The repo is well-documented with strong test coverage, but the lack of a lockfile, missing build instructions for the llama.cpp integration, and single-platform focus limit it to a niche audience. If you're running llama.cpp on Apple Silicon with large context needs, this is worth evaluating; otherwise it's a promising research prototype awaiting broader platform support.