The Problem
Managing multiple LLM providers and APIs is a nightmare. Different endpoints, varying formats, obscure pricing models—it’s a mess. If you’re trying to call OpenAI and Anthropic and HuggingFace and a half-dozen others in one project, good luck. Oh, and you wanted cost tracking, load balancing, and logging? Forget it—this kind of orchestration is usually custom-built and brittle.
What This Does
litellm is essentially middleware for calling 100+ LLM APIs. It offers a Python SDK (cookbook/anthropicagentsdk/main.py is one example) and a proxy server (cookbook/litellmproxyserver) that standardizes API calls into OpenAI’s familiar format. You get consistent endpoints like /chat/completions, /embeddings, and /audio that work across multiple providers.
Beyond basic API calls, litellm handles load balancing, cost tracking, and guardrails. Want to monitor token usage or fallback to another model if your primary one fails? The proxy server (cookbook/litellmrouter) has you covered. You can deploy using Docker (cookbook/litellm-ollama-docker-image/Dockerfile) or jump into the Python SDK with a simple pip install. There's even support for agent protocols like A2A via litellm.a2aprotocol, though that’s niche unless you’re into multi-agent workflows.
Real-World Use
Let’s say you’re building a chatbot. You start with OpenAI’s GPT-4 but realize it’s too expensive for low-priority queries. With litellm, your backend can route simple requests through a cheaper provider like Cohere or HuggingFace using cookbook/litellmrouter/loadtestrouter.py. Meanwhile, cost tracking is handled automatically, so you can analyze usage later using the Grafana dashboards (cookbook/litellmproxyserver/grafanadashboard).
Here’s how you’d call Anthropic’s Claude via the SDK:
Or, if you prefer the proxy server:
The Bottom Line
litellm is perfect for teams juggling multiple LLM providers or needing enterprise-grade features like rate-limiting, virtual keys, and logging. But it’s overkill for small projects where you’re just calling one or two APIs. The setup involves a lot of moving parts (e.g., Docker, YAML configs), but if you’re scaling up, it’s worth it.