The Problem
Running open-source LLMs as a production-grade API endpoint is more work than it looks. You need to choose an inference backend, handle model downloading and versioning, expose an OpenAI-compatible interface, and figure out deployment to GPU infrastructure. OpenLLM packages all of that into a single CLI so teams can skip the integration work and get a working server quickly.
What This Does
OpenLLM is a Python CLI that runs open-source LLMs (DeepSeek, Llama, Qwen, etc.) as OpenAI-compatible API endpoints. The core logic lives in src/openllm/ with clear module separation: local.py handles local serving, cloud.py handles BentoCloud deployment, model.py and repo.py manage model resolution and custom repositories, and venv.py handles environment isolation.
The entry point is src/openllm/main.py, which wires the CLI commands. The project uses pyproject.toml for packaging and uv.lock for dependency locking. It supports a broad model catalog—the README lists 15+ models with their required GPU configurations, from a 2B Gemma on 12GB to a 671B DeepSeek needing 16x80GB.
How To Use It
Setup: Install via pip or uv, per the README and pyproject.toml.
pip install openllm
Running it: Start a server with the serve command, specifying model and version.
openllm serve llama3.3:70b
Configuration: No required environment variables for basic use. Custom models are added via a model repository, documented in the README under "Set up a custom repository." Cloud deployment uses BentoCloud, configured through the cloud.py module.
Missing evidence: The README mentions a hello command for interactive exploration (openllm hello). GPU requirements per model are documented in the README table but not enforced by the CLI itself.
Real-World Use
A team prototyping a RAG application needs a local LLM endpoint without committing to a cloud provider. They run openllm serve qwen2.5:7b, get an OpenAI-compatible endpoint at localhost:3000, and point their existing OpenAI SDK code at it by changing the baseurl.
from openai import OpenAI
client = OpenAI(baseurl="http://localhost:3000/v1") response = client.chat.completions.create( model="qwen2.5:7b", messages=[{"role": "user", "content": "Summarize this document."}] )
When the prototype needs scale, the same config deploys to BentoCloud without rewriting the serving layer.
Code Health & Issues
Low - No lockfile for reproducibility: pyproject.toml declares dependencies but uv.lock is present, which mitigates this. The lockfile is committed, so builds are reproducible with uv. Verify that CI uses uv rather than plain pip. Low - Minimal test coverage: Only 1 test file for a project with 12 Python source files. The serving, model resolution, and cloud deployment paths are largely untested. Low - CI relies on pre-commit.ci: The badge in the README points to pre-commit.ci, not a self-hosted CI pipeline. The .github/workflows/ directory contains release and dependabot automation, but no visible test workflow. Low - README is template-driven: README.md is generated from README.md.tpl via gen_readme.py. Keep edits in the template, not the generated file, or they'll be overwritten.
The Bottom Line
OpenLLM is a practical tool for teams that want to run open-source LLMs without building serving infrastructure from scratch. The model catalog is current, the OpenAI-compatible API removes integration friction, and the BentoCloud path handles production deployment. The thin test suite and reliance on external CI are the main concerns; treat it as a solid foundation rather than a finished product. Best suited for teams already in the BentoML ecosystem or those needing a quick, consistent LLM serving layer across local and cloud environments.