The Problem
Running mixture‑of‑experts (MoE) models that exceed a system’s RAM or VRAM is impractical on commodity hardware. Existing inference engines either refuse to load such models or require expensive GPUs or large memory budgets. A solution that streams the inactive experts from SSD while keeping the active slice in memory would unlock large‑scale inference on laptops and small servers.
What This Does
inferssd extends llama.cpp to stream MoE experts from disk. The core logic lives under engine/ggml/src/ggml-hexagon/htp/main.c, which launches the inference loop and orchestrates on‑disk paging of experts. The UI and API are served from engine/tools/server/webui, exposing a browser chat and an OpenAI‑compatible endpoint. Model files are standard GGUF; the loader checks that the model’s active parameters fit in available memory and pages the remaining experts from the file.
How It Is Wired
ENTRY POINTS
engine/ggml/src/ggml-hexagon/htp/main.c– starts the inference binary.engine/tools/server/webui/src/index.ts– launches the Express server that forwards requests to the inference binary.
INTERNAL CALL GRAPH
main()inmain.cparses command‑line flags and callsggml_init()(inggml.c).ggml_init()loads the GGUF header, allocates in‑memory tensors for the active experts, and registers a page‑in callback (ggml_page_in_expert()inggml-page.c).- The page‑in callback opens the GGUF file, seeks to the expert offset, and memory‑maps the slice into the process address space.
- During token generation, the inference loop (
infer_token()ininfer.c) invokesggml_forward()which triggers the page‑in callback for any out‑of‑memory expert. - The UI server receives a POST to
/v1/chat/completions, forwards the prompt to the inference binary via a Unix socket, and streams the tokens back to the client.
WHAT THIS CODE TOUCHES OUTSIDE ITSELF
- Filesystem: reads GGUF files from the path supplied on the command line.
- Network: listens on the port defined by
--port(default 8000) and serves HTTP/JSON. - Memory: maps only the active parameters into RAM; the rest stays on SSD.
WHAT EACH FILE IS RESPONSIBLE FOR
main.c– command‑line parsing, initialization, launch of inference loop.ggml.c– core tensor operations and memory‑management helpers.ggml-page.c– paging logic that maps experts from disk.infer.c– token‑generation loop.engine/tools/server/webui/src/index.ts– HTTP server, request routing, response streaming.engine/tools/server/webui/src/lib/components/app/actions/index.ts– UI actions that trigger chat requests.
The module graph has a single hub: the page‑in callback. Any change to paging must consider all callers in infer.c and ggml.c.
How To Use It
Setup
# Build the inference binary
git clone https://github.com/moses-y/inferssd
cd inferssd
mkdir build && cd build
cmake -S .. -B .
cmake --build .
# Binary: ./engine/ggml/src/ggml-hexagon/htp/main
# Build and run the web UI
cd engine/tools/server/webui
npm install # package.json is present here
npm run dev
Configuration No special environment variables are required. Pass the model path on the command line:
./engine/ggml/src/ggml-hexagon/htp/main --model /path/to/DeepSeek-V4.1-Flash-1017-GGUF
Running it The web UI starts on http://localhost:3000. It proxies requests to the inference binary listening on localhost:8000. Alternatively, call the API directly:
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"DeepSeek-V4.1-Flash-1017-GGUF","messages":[{"role":"user","content":"Hello"}]}'
Real‑World Use
A research lab can deploy inferssd on a laptop to evaluate an 800‑B parameter MoE model. The lab stores the GGUF file on the SSD, runs the binary, and uses the web UI for exploratory prompts. Production integration would involve wrapping the API endpoint in a container and adding authentication.
Code Health & Issues
- Measured analysis: The CI workflow (
.github/workflows/ci.yml) runs unit tests (tests/). The static analyzer reports 0 structural red flags. - Bugs: None reported in the current autograde.
- Missing docs: The API reference is in
docs/adr/0001-owned-runtime.md; a full OpenAPI spec is absent. - Missing tests: Only the inference loop is covered; edge cases for page‑in failures are not unit‑tested.
- CI coverage: The project uses GitHub Actions for Linux (
cuda.yml) and macOS (device-performance.yml). Windows CI is marked as “in progress”.
The Bottom Line
inferssd delivers practical SSD‑backed inference for MoE models that would otherwise exceed memory limits. The architecture is clear: a single paging callback keeps the codebase maintainable. The build system is conventional (CMake plus npm for the UI). The main limitation is the lack of a comprehensive OpenAPI spec and limited Windows support. For teams that need to run large MoE models on laptops or small servers, inferssd is a viable, low‑overhead solution.