The Problem

Running open-source LLMs (DeepSeek, Llama, Qwen) as production APIs typically means stitching together model download, inference backends, GPU configuration, and OpenAI-compatible serving. OpenLLM collapses that into a single CLI command that serves a model locally or deploys it to a cloud target.

What This Does

This is a fork of bentoml/OpenLLM (12.5k stars upstream). It's a Python CLI (src/openllm/__main__.py) that manages model repositories, serves models as OpenAI-compatible endpoints, and deploys to cloud targets. The package uses Typer for the CLI and wraps BentoML-style model management.

The core modules are src/openllm/common.py (config, env vars, output), src/openllm/local.py (serve/run), src/openllm/cloud.py (deployment), and src/openllm/repo.py (model repo management). A uv.lock file exists, so builds are reproducible despite the heuristic flag in the analysis.

How It Is Wired

Execution starts at src/openllm/__main__.py. The CLI exposes hello, serve, run, and deploy commands. run reaches 33 functions, serve reaches 30. The dispatch goes through _select_action (reaches 10 functions), which routes to either local or cloud paths.

Local serving (src/openllm/local.py) builds the serve command via _get_serve_cmd, then _run_model calls output 10 times — that's the most-called function in the repo. common.py's output is called from 26 places total, making it the widest blast radius: change its signature and you break most of the CLI.

Cloud deployment flows through src/openllm/cloud.py → ensure_cloud_context → _get_deploy_cmd, which runs a subprocess via run_command. The shortest external path is _select_action → get_cloud_machine_spec → subprocess.check_output — 2 hops from entry to leaving the process.

File responsibilities:

  • common.py — config, env vars, output; 38 functions, 9 classes; the hub
  • repo.py — model repo CRUD (cmd_list, cmd_add, cmd_update, cmd_remove)
  • cloud.py — cloud context and deploy command construction
  • local.py — local serve/run, calls the model for inference
  • model.py — model registry lookup (ensure_bento, list_model)
  • venv.py — per-model virtualenv management
  • clean.py — cache/venv/repo cleanup
  • accelerator_spec.py — GPU memory parsing and can_run checks

How To Use It

git clone https://github.com/moses-y/OpenLLM
cd OpenLLM
uv sync  # or pip install -e .
openllm serve llama3.3:70b
openllm deploy llama3.3:70b  # to cloud

Configuration lives in pyproject.toml and src/openllm/common.py (the Config class). Cloud deployment requires a BentoCloud context set via openllm cloud commands.

Real-World Use

A team wanting a self-hosted OpenAI-compatible endpoint for an internal tool:

openllm serve deepseek:r1-671b
# Then point any OpenAI SDK client at http://localhost:3000/v1

Code Health & Issues

Static analysis found 2 medium and 1 low issues:

  • Medium — High branching density in release.sh: 26 branch points over 86 lines. Fix: decompose decision-heavy logic.
  • Medium — No dependency vulnerability scan in CI (.github/workflows). Add dependency-review-action or osv-scanner.
  • Low — No timeout-minutes on workflow jobs in create-releases.yml; a wedged job runs to the 6-hour platform default.

The import graph shows no circular dependencies, and the module structure is clean — 12 modules, 0 import edges between internal modules. The release.sh branching is the main maintainability concern.

The Bottom Line

This is a well-structured CLI wrapper around BentoML's model serving. The fork status (0 stars) means you're relying on upstream maintenance, not this fork's activity. The code is clean and the wiring is simple — good for teams wanting OpenAI-compatible LLM serving without building it themselves. The lack of dependency scanning in CI is the most actionable gap.