The Problem

Practitioners need reproducible, bite‑sized examples that illustrate every layer of LLM inference – from tokenization to multi‑GPU autoscaling. Without a ready‑made portfolio, teams spend days re‑implementing basic building blocks and lose time on environment churn.

What This Does

The repository is a portfolio of 26 independent notebooks, each paired with a short README.

  • Early days (day01‑day08) focus on runtime fundamentals: tokenization (day02/02-b-tokenization.ipynb), KV‑cache mechanics (day05/kv-cache.ipynb), and CUDA kernels (day07/cuda-kernels.ipynb).
  • Mid‑stage notebooks (day09‑day16) explore advanced kernels (vLLM, SGLang), quantization (day14/quantization-gptq-awq-smoothquant.ipynb), and speculative decoding (day15/speculative-decoding.ipynb).
  • Later days (day19‑day26) address infrastructure: GPU architecture (day19/gpu-architecture-sms-hbm.ipynb), containerization (day22/containerization-docker-nim.ipynb), autoscaling (day23/autoscaling-concurrency-cold-starts.ipynb), and zero‑downtime deployment (day26/zero-downtime-deployment.ipynb).

Common code patterns (e.g., softmax, layer_norm, split_heads, multi_head_attention) appear across notebooks, forming a reusable inference kernel library.

How It Is Wired

The only explicit entry point discovered is the cell run in day16/kv-cache-prefix-caching.ipynb (line 204). That cell invokes two functions, which in turn call the shared core:

  • softmax – called from 10 locations (e.g., speculative_decode_real, simulate_acceptance_rate).
  • layer_norm – called from 8 locations (e.g., prefill, transformer_block_no_cache).

Typical execution flow in a notebook:

  1. Setup – import torch, numpy, and any local helpers defined in the same notebook.
  2. Model loading – a single function (often encode → bpe_merge → get_pairs) parses a checkpoint or safetensors file.
  3. Cache preparation – KVCache class is instantiated (used in day05, day09, day16).
  4. Inference loop – calls multi_head_attention → split_heads → softmax → layer_norm.
  5. Metrics – functions such as simulate_acceptance_rate or ttft_ms write CSV/PNG files locally.

All side effects are confined to the notebook’s working directory: file reads/writes, a single outbound HTTP call to fetch model weights, and occasional subprocess launches for profiling. No persistent service is started, and no cross‑notebook state is shared.

Because the call graph is limited to intra‑notebook functions, the blast radius of a change is predictable: modifying softmax impacts up to ten notebooks; altering KVCache touches three notebooks that instantiate the class.

How To Use It

# 1. Clone the repository
git clone https://github.com/moses-y/100-days-of-inference.git
cd 100-days-of-inference

# 2. Install a matching Python environment manually (e.g., Python 3.11, PyTorch 2.3)
#    The repo does not provide a requirements file; install the basics yourself:
pip install torch==2.3.0 transformers==4.40.0 tqdm==4.66.0

# 3. Open a notebook of interest, e.g.:
jupyter notebook day04/attention.ipynb

If a notebook references a large binary (e.g., Inference Engineering.pdf), ensure the file is present in the repository root; otherwise the notebook will raise FileNotFoundError.

Real‑World Use

A team building a custom inference service could copy day05/kv-cache.ipynb to prototype a KV‑cache layer, replace the toy model loading with their own checkpoint, and then extract the KVCache class into a library module. The surrounding notebooks provide ready‑made benchmarks (day06/ops-byte-ratio.ipynb) to validate latency and throughput before integration.

Code Health & Issues

  • HIGH – Pin the environment – 27 notebooks lack any requirements.txt or environment.yml. Without a manifest the code cannot be reproduced reliably.
  • MEDIUM – Large binary in repo – Inference Engineering.pdf (23 MB) exceeds typical Git size limits; move it to Git LFS or external storage.
  • MEDIUM – Missing random seed – Stochastic notebooks such as day06/ops-byte-ratio.ipynb do not set a fixed seed, making results nondeterministic.
  • MEDIUM – No CI/CD pipeline – No .github/ workflows, Dockerfile, or other automation; adds manual overhead for testing.
  • MEDIUM – No LICENSE – The repository does not declare usage rights, creating legal ambiguity for downstream users.

No test suite beyond a single test file is present, and there are no lockfiles (requirements.txt, poetry.lock, etc.).

The Bottom Line

The repo offers a thorough, day‑by‑day showcase of LLM inference techniques, useful as a learning sandbox or rapid prototyping source. Its main drawbacks are the lack of reproducible environment specifications, missing CI, and an oversized PDF that bloats the repo. Teams that need reproducibility should first create a consolidated requirements.txt and migrate large assets to LFS before adopting the notebooks in production pipelines.