The Problem

Running inference with a multi‑trillion‑parameter model typically requires GPUs, large memory footprints, and heavyweight frameworks. Teams that need a portable, CPU‑only deployment—e.g., on edge servers or low‑cost laptops—have no lightweight reference implementation that fits inside a few gigabytes of RAM.

What This Does

kimi-k3-in-c provides a pure C99 inference engine for the 2.78‑trillion‑parameter Kimi K3 model. The binary (./bin/k3) streams expert weights from disk, keeps a 8 GB resident working set, and produces byte‑identical results across memory budgets. Core source lives under src/:

  • src/cli/k3_run.c – the main entry point and CLI flag parsing.
  • src/core/k3_ops.c – high‑level model operations (tokenisation, KV cache handling).
  • src/cache/k3_cache.c – implements the three‑phase cache described in docs/images/cache-3phase.mmd.

Supporting headers in include/k3/ expose the public API (k3.h, k3_cfg.h). Utilities such as tools/cmp_logits.py and benchmarks/bench_kernels.c aid validation and performance measurement.

How It Is Wired

Execution starts in src/cli/k3_run.c → int main(int argc, char **argv). The CLI parses options (model path, trunk directory, preset, prompt, generation length) and builds a k3_cfg_t struct defined in include/k3/k3_cfg.h. This configuration is passed to k3_run(&cfg) (implemented in src/core/k3_ops.c).

k3_run performs:

  1. Model loading – calls k3_load_trunk() (in src/core/k3_ops.c) which memory‑maps the compressed trunk files from --trunk.
  2. KV cache allocation – invokes k3_cache_init() from src/cache/k3_cache.c.
  3. Generation loop – repeatedly calls k3_decode_step() (core op) which fetches expert weights on‑the‑fly, runs the quantised kernels (found in src/kernels/ – not listed but referenced), and updates the cache.

All I/O is confined to the binary’s own directory tree; no network or database calls occur. The Python tools (tools/*.py) import no internal modules (import graph shows 0 edges), so they operate independently as scripts that read model dumps or compare logits. No circular dependencies are present, keeping the C call graph shallow.

How To Use It

# 1. Clone the repo (URL must be used verbatim)
git clone https://github.com/moses-y/kimi-k3-in-c.git
cd kimi-k3-in-c

# 2. Build the engine (Makefile target)
make -j$(nproc)          # produces ./bin/k3

# 3. (Optional) Download a model checkpoint – see scripts/download-model.sh
./scripts/download-model.sh   # fetches ~/k3model and ~/k3trunk

# 4. Run a quick inference (no model needed for a smoke test)
./bin/k3 --help

# 5. Full inference example (requires the checkpoint)
./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
      --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental

The --preset flag selects a memory budget (e.g., laptop, server) defined in docs/QUICKSTART.md. No environment variables are required; all configuration is supplied via CLI flags.

Real‑World Use

A data‑science team can embed the binary in a nightly batch job that scores large text corpora on modest VMs. The job script would:

#!/usr/bin/env bash
MODEL=~/k3model
TRUNK=~/k3trunk
for txt in data/*.txt; do
  prompt=$(head -n1 "$txt")
  ./bin/k3 "$MODEL" --trunk "$TRUNK" --preset laptop \
        --prompt "$prompt" --gen 64 >> results/$(basename "$txt").out
done

Only the first line of each document is streamed; the rest of the model stays on disk, keeping RAM under 10 GB.

Code Health & Issues

  • MEDIUM – Large binary (tests/fixtures/tiny_k3.bin, 8.5 MB) stored in Git. → Move to Git LFS or external storage.
  • MEDIUM – GitHub Actions checkout keeps the token (.github/workflows/ci.yml). → Add persist-credentials: false.
  • LOW – No job timeout defined (.github/workflows/ci.yml). → Set timeout-minutes to bound CI runs.

Additional static findings:

  • HIGH – Deep nesting (max depth 10) in src/cache/k3_cache.c and src/cli/k3_run.c → refactor with early returns.
  • HIGH – Duplicated 6‑line blocks across benchmarks and tests → extract shared helpers.
  • MEDIUM – Files opened without context manager in three Python tools → wrap with with open(...).
  • MEDIUM – Broad except: clauses in tools/cmp_logits.py → catch specific exceptions.
  • MEDIUM – Oversized source files (src/cli/k3_run.c, src/core/k3_ops.c) → split by responsibility.
  • MEDIUM – High branching density in several benchmark scripts → replace with table‑driven dispatch.

No critical or high‑severity issues were reported.

The Bottom Line

kimi-k3-in-c delivers a verifiable, CPU‑only inference path for a 2.78 T‑parameter model within 8 GB RAM, making it uniquely suited for low‑resource environments. The codebase is functional but suffers from deep nesting, duplicated snippets, and a few CI hygiene gaps that should be addressed before production use. Teams comfortable with C and willing to clean up the identified hotspots can adopt it as a lightweight alternative to GPU‑bound large‑model runtimes.