The Problem
Running LLM inference on Kubernetes at production scale means fighting the same battles repeatedly: routing requests to the right replica, keeping KV cache hot across multi-turn conversations, serving models too large for one accelerator, and keeping multi-tenant traffic stable. Each team re-solves these by hand. llm-d packages the solutions as tested, benchmarked recipes so you don't rediscover them.
What This Does
llm-d is a CNCF sandbox project (founded by Red Hat, Google Cloud, IBM Research, CoreWeave, NVIDIA) that sits above model servers like vLLM and SGLang. It provides orchestration and optimization patterns for intelligent routing, tiered KV-cache management, prefill/decode disaggregation for huge models, and SLO-aware autoscaling. The bulk of the repo is guides/ (740 files) — "well-lit path" recipes with Helm charts and benchmarks — plus docker/ (37 files) building CPU, CUDA, ROCm, and XPU container images.
This is a fork (moses-y/llm-d) of the upstream project with 4252 stars. It is not a library you import; it is a collection of deployable patterns, CI pipelines, and Docker build scripts.
How It Is Wired
There is no central application entry point. Execution starts in the GitHub Actions workflows (.github/workflows/, 142 files) — nightly e2e runs like nightly-e2e-pd-disaggregation-gke-acc-gpu-vllm-x.yaml invoke shell verification scripts under .github/scripts/e2e/ and .github/scripts/nightly-e2e-verification/. Those scripts (e.g. verify.py, verify_helpers.py) drive cluster checks and report results via consolidate-status-*.yaml workflows.
The import graph shows 16 internal Python modules with 0 import edges — the Python code is effectively disconnected, not a package. The real "wiring" is shell orchestration and YAML. docker/scripts/cuda/builder/*.sh (build-ucx, build-nvshmem, build-nixl) are the deepest logic, assembling accelerator libraries into images. The widest blast radius is scripts/guide.py (1133 lines), the oversized generator for guide content — a change there ripples across every generated guide.
How To Use It
The README points to the documentation site (llm-d.ai/docs/well-lit-paths) for recipes and Helm charts, but the excerpt documents no explicit install commands. Evidence present: a Makefile, docker/Dockerfile.cpu/.cuda/.rocm/.xpu, and requirements.txt files in guide subdirectories. Clone with git clone https://github.com/moses-y/llm-d, then build an image (e.g. docker build -f docker/Dockerfile.cuda .) or follow a guide's Helm instructions. A lockfile is absent, so dependency installs are not reproducible.
Real-World Use
A team serving Llama 3.1 70B on AMD MI300X deploys the optimized-baseline guide's Helm chart, adds the prefix-cache-aware router from the intelligent-routing theme, and the README's cited benchmarks show 3x output throughput and 2x faster TTFT versus round-robin. For a multi-tenant cluster, the flow-control guide protects tenants while the SLO-aware autoscaling templates (in guides/workload-autoscaling/slo-aware/) scale on inference signals.
Code Health & Issues
Static analysis found 22 issues (4 high, 18 medium) across 6 kinds. High: deep nesting (max indentation depth 8) in e2e-validate-scale-event.sh, verify_helpers.py, and plot_run.py; and 111 duplicated 6-line blocks across 16 shell files. Medium: broad exception handling in plot_run.py and lint-dockerfile-envvars.py, high branching density (49 branch points over 162 lines in e2e-validate-scale-event.sh), and the oversized scripts/guide.py.
The CI audit adds: High — third-party GitHub Actions pinned to mutable tags (dorny/paths-filter@v4, etc.), a known secret-leak vector; High — 5 test files exist but no workflow invokes them; High — continue-on-error on a correctness step in build-image.yaml line 188. Medium — ubuntu:22.04 base image unpinned by digest in Dockerfile.cpu; no non-root USER. Low — workflow jobs lack timeout-minutes. Tests and CI exist; no committed secrets; license present.
The Bottom Line
The guide content and benchmark data are the real value — genuinely useful, vendor-backed recipes. The code surface is thin and the CI has real correctness gaps (unpinned actions, tests never run). Use it as a reference for production inference patterns, not as a codebase to extend.