The Problem

Local LLM inference requires balancing compression ratio against latency and output quality. TurboQuant+ addresses this by achieving 4.6x KV cache compression with q8_0 speed parity on Apple Silicon, but the codebase shows architectural debt that affects maintainability and ripple effects when modifying core quantization logic.

What This Does

TurboQuant+ implements TurboQuant (ICLR 2026) KV cache compression using PolarQuant + Walsh-Hadamard rotation, achieving 4.6x compression with only 1% quality loss and q8_0 speed parity on M5 Max. The repo contains 61 files across three projects (turboquant, benchmarks, scripts) with Python as the primary language. Key performance data from the README shows turbo3 at 2747 prefill tok/s vs q8_0's 2694 tok/s, with needle-in-haystack retrieval at 80% (16/20) across 4K-32K contexts. The code health audit found 15 measured issues across 6 severity levels, including missing lockfile, broad exception handling in 3 files, and duplicated 6-line blocks repeated 19 times across 6 benchmark/test files.

How It Is Wired

Execution starts at main in benchmarks/run_benchmark.py:173, which reaches 124 functions and is itself called from 14 places. The internal call graph shows 1154 resolved call edges; _make_log is the most widely called function at 107 call sites. Key entry-point paths: main -> run_benchmark -> api_request [subprocess via Popen] and main -> run_benchmark -> wait_for_server. The most connected modules are turboquant/__init__ (Ca=4, Ce=4, instability=0.5) and turboquant/polar_quant (Ca=5, Ce=2, instability=0.29). Files with the widest blast radius: tests/test_turbo_hardware_diag.py (298 functions, 31 classes, called from 2 files), tests/test_niah.py (106 functions, 28 classes), and scripts/turbo_hardware_diag.py (86 functions, 4 classes, runs external commands, reads/writes files, makes outbound network calls). The call graph reveals _make_completed_process called from 22 places and quantize from 26 places as major branching points.

How To Use It

Setup: No lockfile is committed (per pyproject.toml), so reinstall with uv pip install -e . or pip install -e . to generate the lockfile. The package has no Dockerfile; build runs via GitHub Actions CI.

Configuration: No environment variables or keys are required for core functionality. The README documents --cache-type-k turbo3 --cache-type-v turbo3 for Apple Silicon Metal support.

Running it: Invoke the benchmark entry point:

python -m benchmarks.run_benchmark --cache-type-k turbo3 --cache-type-v turbo3

Or run the hardware diagnostic script:

python scripts/turbo_hardware_diag.py

Real-World Use

In a deployment pipeline compressing KV cache for a 35B MoE model on M5 Max, turbo3 achieves 4.6x compression with only a 2% decode speed reduction versus q8_0 (78.4 vs 85.2 tok/s at short context). The quantization pipeline flows through turboquant/polar_quant.py (quantize/dequantize functions), then turboquant/outlier.py (compress_and_compare), with outlier handling via turboquant/outlier.py's _compute_channel_split. For a Qwen3.5-35B-A3B MoE, this enables 3-bit KV cache storage at 128GB RAM while preserving needle-in-haystack retrieval at 80% across 32K context.

Code Health & Issues

  • [HIGH] Commit a lockfile beside the manifest - pyproject.toml has no lockfile, creating non-reproducible builds where tested artifacts may differ from shipped artifacts.
  • [HIGH] Deep nesting x6 - turboquant/turboquant.py, turboquant/hw_replay.py, scripts/turbo_hardware_diag.py have max indentation depth 7; control flow is hard to follow.
  • [HIGH] Duplicated code blocks - 19 repeated 6-line blocks across benchmarks/demo.py, benchmarks/test_with_llama.py, tests/test_hw_replay.py, tests/test_turbo_hardware_diag.py; extract shared helpers.
  • [MEDIUM] Broad exception handling x3 - benchmarks/run_benchmark.py, scripts/niah_test.py, scripts/turbo_hardware_diag.py have bare/Exception-wide except that swallows errors indiscriminately.
  • [MEDIUM] Oversized files x4 - scripts/niah_test.py, scripts/turbo_hardware_diag.py, tests/test_turbo_hardware_diag.py have 1333 code lines; hard to hold in one head with wide ripple effects.
  • [LOW] No lockfile committed - standard Python SDLC gap; see HIGH finding.

The Bottom Line

This repo delivers on its core promise: 4.6x KV cache compression with q8_0 speed parity on Apple Silicon, validated across context lengths and quality metrics. However, the codebase carries significant technical debt—missing lockfile, duplicated logic, and deeply nested control flow—that increases risk for any production modifications. Teams needing stable, maintainable KV cache compression should pin to a locked dependency set and allocate refactoring time for the nested control flow and duplicated blocks before extending the system.