model serving minefield

The Problem

Engineers repeatedly waste hours chasing “wrong numbers” that stem from subtle serving‑stack bugs (template mismatches, quant‑kernel paths, memory‑allocation quirks, etc.). The repository collects concrete, reproducible traps so teams can spot the root cause before a measurement is trusted.

What This Does

The repo is a registry of LLM‑serving failure patterns. Each trap lives as a markdown file under traps/ (e.g., traps/quantization/10-quant-label-is-not-the-kernel-path.md) and is paired with a check that can detect the condition automatically.

The checks live in checks/ (e.g., checks/preflight_template.py, checks/cache_hit_probe.py). They implement lightweight validators that inspect a running service, a request/response payload, or a system configuration and emit a verdict.

The doctor (doctor/minefield_doctor.py) orchestrates the checks, fetches remote templates, and renders a unified report that maps symptoms to the offending trap entry.

How It Is Wired

Execution starts at two concrete entry points:

Typical flow for a check

1. main parses CLI arguments and calls run_checker (via the internal call graph, 43 callers).
2. run_checker invokes a specific check function such as check_reasoning_fields.
3. The check may perform a network request (urllib.request.Request in doctor/minefield_doctor.py) or launch an external command (subprocess.call in checks/cache_hit_probe.py).
4. Results are written to the filesystem (open calls in various modules) and aggregated into a JSON verdict.

Key modules and blast radius

• doctor/minefield_doctor.py – 47 functions, makes outbound HTTP calls and runs external commands; it is the hub for all checks, so changes here affect every trap detection.
• checks/cache_hit_probe.py – 7 functions, includes the most external calls (network + subprocess) and is invoked from four other files.
• integrity/claim_propagation.py – 13 functions, read by 10 callers; handles ledger validation and is a common utility.

The import graph shows no circular dependencies and zero import edges between the 39 internal modules, indicating a deliberately flat package layout. The internal call graph contains 763 resolved edges, but the most‑connected modules (FixtureLane, diagnose, run_checker) are all confined to the doctor/ and checks/ directories, keeping the core logic isolated.

External interactions

• 8 functions perform network I/O (e.g., run -> check_streaming).
• 20 functions invoke shell commands via subprocess.
• 14 functions read/write files (mostly in test helpers and the doctor’s report writer).

No persistent secrets are stored in the repo.

How To Use It

# 1. Clone the repo
git clone https://github.com/moses-y/model-serving-minefield
cd model-serving-minefield

# 2. Install Python requirements (the repo uses `requests` and `transformers`)
#    A requirements file is not present; install manually:
pip install requests transformers

# 3. Run a specific check (example: cache‑hit probe)
python checks/cache_hit_probe.py --target http://localhost:8000

# 4. Or run the full doctor suite to audit all registered traps
python doctor/minefield_doctor.py

Configuration – The doctor reads remote template URLs defined inside its own functions (_req, get, post). No external config files are required for a basic run.

Running tests – Test files live under integrity/tests/ and checks/tests/. They can be executed with pytest once the dependencies above are installed.

Real‑World Use

A serving platform team integrates the doctor into their CI pipeline:

# .github/workflows/integrity.yml (add a test step)
- name: Run Model‑Serving Minefield checks
  run: python doctor/minefield_doctor.py

When a new container image is built, the workflow automatically flags any quant‑kernel mismatch or hidden‑state misalignment before the service is promoted to production, turning a multi‑day debugging session into a fast CI failure.

Code Health & Issues

• HIGH – CI never executes the test suite (.github/workflows/* lacks a test step).
• MEDIUM – GitHub token permissions are not scoped (.github/workflows/integrity.yml).
• MEDIUM – Outbound HTTP calls lack a timeout (doctor/minefield_doctor.py).
• LOW – Workflow jobs have no timeout‑minutes limits (.github/workflows/integrity.yml).

Additional static findings (19 high, 29 medium) include duplicated 6‑line blocks across 25 files, deep nesting (max indentation depth 10), oversized files (doctor/minefield_doctor.py ≈ 1900 lines), un‑guarded open() calls, broad exception catches, and high branching density. The repository otherwise includes a licence, tests, and a functional CI pipeline.

The Bottom Line

Model‑Serving‑Minefield provides a well‑organized, test‑backed catalog of LLM‑serving pitfalls and a thin Python driver that can be dropped into any CI/CD flow. The codebase is small, dependency‑light, and free of circular imports, but it suffers from poor CI coverage, missing token‑scoping, and a few maintainability hotspots (large monolithic files and duplicated helpers). Teams that need systematic guardrails against serving‑stack bugs will find it immediately useful; they should tighten the CI workflow and refactor the oversized doctor module for long‑term scalability.