The Problem

Running a 26‑billion‑parameter LLM on an Apple‑silicon Mac with only 8 GB of RAM is normally impossible because the full weight set (≈14 GB) must reside in memory. Developers need a way to stream the model’s sparse expert weights from SSD and keep only a small active core in RAM, while still delivering usable token‑per‑second throughput.

What This Does

TurboFieldfare implements a Swift + Metal runtime that partitions the Gemma 4 26B‑A4B model into a shared 1.35 GB core, a 4 KB FP16 KV cache, and 4‑bit‑quantised expert tables stored on disk. The Sources/TurboFieldfare/Runtime/Inference/Model.swift class orchestrates loading the core and delegating per‑token expert fetches to the streaming subsystem (Infrastructure/Streaming/ExpertStreamer.swift). Metal kernels in Sources/TurboFieldfare/Kernels/* execute the heavy matrix ops directly on the GPU, avoiding CPU bottlenecks. A native macOS app (TurboFieldfareApp/Mac/App/TurboFieldfareMacApp.swift) and a CLI (TurboFieldfareCLI/Command/main.swift) both invoke the same runtime, so the same low‑memory inference path is available for interactive use or scripting.

How It Is Wired

  • Entry point – The CLI and app start at Sources/TurboFieldfareCLI/Command/main.swift (or the macOS app’s TurboFieldfareMacApp.swift). Both instantiate AppInferenceClient (app) or call Generator (CLI) which creates a Model object.
  • Model construction – Runtime/Inference/Model.swift reads the manifest (Infrastructure/ModelIO/ManifestReader.swift) and creates a ModelExpertIO that knows which expert files live on disk.
  • Streaming – When generating a token, ModelExpertIO asks Infrastructure/Streaming/ExpertStreamer.swift for the required expert weights. The streamer opens the repacked model file (Repack/) and reads only the needed 4‑bit blocks, handing them to the Metal context (Infrastructure/Metal/MetalContext.swift).
  • GPU execution – The Metal kernels (e.g., Kernels/Attention/Attention.swift, Kernels/Prefill/PrefillPerHeadNorm.swift) are compiled from the .metal files under Sources/TurboFieldfare/Metal/. MetalContext uploads the core weights and KV cache, then dispatches the kernels.
  • Result flow – After the kernels finish, Generator collects the logits, passes them to Sampler (Runtime/Generation/Sampler.swift), which produces a token string via Tokenizer (Tokenization/Tokenizer.swift). The token is returned to the CLI or UI.

Only three files (MetalContext.swift, ManifestReader.swift, PackedExpertsLayout.swift) exhibit deep nesting (max indentation depth 13), which makes the control flow harder to follow. No circular import graph was detected, so the call hierarchy is linear apart from the streaming layer.

How To Use It

# Clone the exact repository
git clone https://github.com/moses-y/turbo-fieldfare
cd turbo-fieldfare

# Build the release binaries (Swift 6.2 required)
swift build -c release

# Run the macOS GUI
.build/release/TurboFieldfareMac

# Or use the CLI directly
.build/release/turbo-fieldfare-cli generate --prompt "Explain quantum tunnelling"

The first launch triggers a model download and repack (Repack/Core/Remote/HuggingFaceRemote.swift), after which the model can be loaded from AppModel.swift. No environment variables or external services are required beyond an internet connection for the initial download.

Real‑World Use

A content‑generation microservice on an M2‑Air can embed the CLI as a subprocess. The service reads a prompt from an HTTP request, calls turbo-fieldfare-cli generate, captures stdout, and returns the token stream. Because the runtime never exceeds ~2 GB RAM, the same VM can host multiple concurrent workers without swapping.

Code Health & Issues

  • High cognitive_load – Deep nesting in Sources/TurboFieldfare/Infrastructure/Metal/MetalContext.swift, ManifestReader.swift, PackedExpertsLayout.swift (max depth 13). Fix: flatten with early returns/guard clauses, extract inner blocks.
  • Low – CI workflow missing explicit timeout-minutes (.github/workflows/ci.yml). Fix: add a reasonable timeout to prevent overlapping runs.
  • Tests (137 files) and CI (GitHub Actions) are present; license (Apache 2.0) is included; no committed secrets detected. No Dockerfile or lockfile, which is acceptable for a Swift‑only project.

The Bottom Line

TurboFieldfare delivers a practical, low‑memory inference path for a 26 B LLM on Apple‑silicon Macs by streaming quantised expert weights and executing core ops on Metal. The codebase is functional and well‑tested but suffers from deeply nested control flow in key infrastructure files, and its CI lacks job timeouts. Engineers comfortable with Swift and Metal can adopt it for on‑device AI workloads where RAM is scarce.