The Problem

Teams that need to experiment with dozens of LLM providers spend weeks writing adapters, normalising request formats, and stitching together cost‑tracking, rate‑limit and logging logic. The result is duplicated code and fragile integrations that break when a provider changes its API.

What This Does

litellm bundles a Python SDK and an AI‑Gateway (proxy server) that expose a single OpenAI‑compatible HTTP surface (/chat/completions, /embeddings, /images, …). The SDK forwards calls to over 100 back‑ends (Bedrock, Azure, Vertex AI, Anthropic, Cohere, etc.) while automatically applying:

  • per‑request cost accounting (litellm/integrations/additional_logging_utils.py),
  • optional guardrails and routing rules (litellm/types/proxy/guardrails/*),
  • multi‑key load‑balancing (litellm/proxy/example_config_yaml/custom_handler.py defines completion / acompletion and a custom LLM class).

The repository is a portfolio of nine self‑contained projects: the core litellm library, a React/Next UI dashboard, enterprise extensions, a set of “cookbook” examples, and supporting scripts. The recurring technique is a thin wrapper around each provider’s native client that translates the request into the OpenAI schema, then hands it to the provider‑specific SDK.

How It Is Wired

Execution start – proxy server

The proxy is launched via the console entry point defined in setup.cfg/pyproject.toml (installed as litellm). Internally it runs litellm/proxy/server.py (not listed but the entry point of the litellm[proxy] extra). The first request hits the FastAPI router, which dispatches to the completion handlers in litellm/proxy/example_config_yaml/custom_handler.py.

  • completion → acompletion → map_openai_params (called 2×) → get_supported_openai_params (called 2×).
  • These functions are the hub of the internal call graph: completion is invoked from 7 distinct call sites, making it the widest‑impact change point.

Provider dispatch

map_openai_params normalises the incoming OpenAI payload and forwards it to the appropriate provider module under litellm/llms/. For example, Azure image generation routes through litellm/llms/azure/vector_stores/transformation.py (get_complete_url, validate_environment).

Outbound actions

  • Network calls – 5 functions (e.g., AdditionalLoggingUtils.async_health_check, locustfile.chat_completion) issue HTTP traffic to provider endpoints.
  • Model inference – 3 functions (e.g., tests in tests/pass_through_tests/test_anthropic_passthrough_basic.py) call the selected LLM.
  • Cryptography – litellm/litellm_core_utils/app_crypto.py encrypts/decrypts JSON payloads for secure key storage.
  • External command – a single function executes a shell command (found in the CI scripts).

Side‑effects and blast radius

  • The custom handler (custom_handler.py) owns the completion/acompletion functions; any modification here propagates to every API endpoint.
  • app_crypto.py is the only place performing encryption; a bug would affect all stored keys.
  • Logging utilities (additional_logging_utils.py) make outbound health‑check calls; failures can delay request processing.

The call graph shows no deep cycles; the longest traced path is ~4 hops from the HTTP entry to the provider client, keeping change impact predictable.

How To Use It

# Clone the repo (use the exact URL as requested)
git clone https://github.com/moses-y/litellm
cd litellm

# Install the SDK + proxy extras
pip install 'litellm[proxy]'

# Optional: copy the example config and edit keys
cp litellm/proxy/example_config_yaml/config.yaml.example litellm/proxy/config.yaml
# Edit litellm/proxy/config.yaml to add your provider API keys

# Run the proxy (default port 4000)
litellm --model=gpt-4o   # starts FastAPI server

# Test with the OpenAI client library
pip install openai
python - <<'PY'
import openai, os
os.environ["OPENAI_API_KEY"] = "any-key"
client = openai.OpenAI(base_url="http://127.0.0.1:4000/v1")
resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role":"user","content":"Hello"}]
)
print(resp)
PY

The UI dashboard lives under ui/ and can be built with npm install && npm run build (Dockerfile present for containerised deployment). Enterprise extensions add team‑level key‑management and quota enforcement (enterprise/).

Real‑World Use

A SaaS platform that offers “pay‑as‑you‑go” LLM access can point its backend to a single litellm proxy. Each tenant receives a virtual API key; the proxy enforces per‑tenant quotas (via key_management_endpoints.py) and logs usage to a central database. Adding a new provider only requires adding its credentials to config.yaml—no code changes.

Code Health & Issues

  • High – Pin GitHub Actions – .github/workflows/* uses @vN; replace with commit SHAs.
  • High – Remove committed .env – ui/litellm-dashboard/.env.development and .env.production contain live secrets. Delete, .gitignore, and provide an example file.
  • High – CI never runs tests – Workflows lack a test step despite 1,968 test files. Add a pytest step.
  • High – Push directly to default branch – publish-migrations.yml pushes without PR. Change to PR workflow or tag‑only push.
  • Medium – Least‑privilege GITHUB_TOKEN – ghcr_helm_deploy.yml declares no permissions. Add contents: read.
  • Medium – No dependency‑vulnerability gate – No scanner in CI. Add dependency-review-action or osv-scanner.
  • Medium – Large binaries in repo – docs/.../litellm_test_connection.gif (16 MB) and other >5 MB files inflate clones; move to Git LFS or external storage.
  • Medium – Checkout persists token – publish-migrations.yml should set persist-credentials: false.
  • Medium – Docker image runs as root – Add a non‑root USER and chown steps.
  • Low – Missing job timeouts – ghcr_deploy.yml lacks timeout-minutes; set reasonable limits.

The repository otherwise includes a full test suite, CI pipelines, a Dockerfile, and a licence file, indicating a mature development process.

The Bottom Line

litellm delivers a practical, OpenAI‑compatible façade over a sprawling LLM ecosystem and includes a ready‑to‑run proxy server. Its modular design and extensive test coverage make it a solid foundation for internal AI gateways. However, the repo ships live secrets, unpinned CI actions, and a few operational hygiene gaps that must be fixed before production use. Teams comfortable managing their own CI/CD and container security will find it valuable; otherwise consider the hosted Enterprise offering.