LiteLLM

LiteLLM is an open-source AI gateway and Python SDK built by BerriAI, a Y Combinator Winter 2023 company founded by Krrish Dholakia and Ishaan Jaffer. The tool solves a specific, concrete problem: as teams add more LLM providers to their stack, each provider ships its own API format, authentication model, and error behavior. LiteLLM sits in front of all of them and translates every call into OpenAI's format, so application code never has to know which provider it is talking to. By April 2026, the project has reached 45,400 GitHub stars, 1,000-plus contributors, and a self-reported 1 billion requests served through its proxy infrastructure.

The tool ships in two modes. The Python SDK lets you call litellm.completion() directly in code and handles format translation, retries, and fallbacks automatically across 100-plus providers including OpenAI, Anthropic, Google Vertex AI, AWS Bedrock, Azure OpenAI, Cohere, Mistral, HuggingFace, NVIDIA NIM, Ollama, and vLLM. The Proxy Server mode is a self-hosted HTTP gateway you deploy via Docker: any existing OpenAI SDK client points to it without code changes, and the proxy handles routing, virtual API key management, per-team budget limits, rate limiting, and observability integrations to tools like Langfuse and Arize Phoenix. Both modes are MIT-licensed and free to run. Enterprise governance features, including SSO, RBAC, and audit logs, are available in paid tiers starting at $250 per month.

What LiteLLM actually does in April 2026

The core function is provider translation. When your application sends an Anthropic-format request through LiteLLM, the library rewrites the input and output fields to match what the target model expects, normalizes error codes across providers, and returns a response in OpenAI's format regardless of which model handled the call. New providers are typically added within a day of their public API release, which is how LiteLLM has kept pace with the industry's rapid expansion beyond 100 supported endpoints.

The proxy server extends this with production-grade access management. Teams configure it with a YAML file that defines model groups, per-key budgets, and routing rules. A platform team can issue virtual API keys to individual squads with hard spending caps, model allowlists, and RPM limits enforced at the proxy layer. When a primary model is unavailable or rate-limited, the proxy falls back to a configured alternative automatically. Load balancing distributes requests across multiple instances of the same provider. All of this happens without the application knowing about routing logic.

Observability integrates externally rather than natively. LiteLLM routes logs to Langfuse, Arize Phoenix, Prometheus, or OpenTelemetry. Teams then visualize cost attribution by team, key, or user in their observability platform of choice. The proxy UI provides a basic spend dashboard, but production monitoring setups generally wire it to external tools for alerting and long-term retention.

"The idea of an LLM proxy is super compelling. It could help teams dynamically choose between local and cloud LLMs without restructuring their infrastructure." -- jmorgan (Ollama creator), Hacker News, December 2023

The Python SDK is often how developers first encounter LiteLLM. The interface is a thin wrapper: install the package, set provider API keys as environment variables, and call litellm.completion(model="gpt-4o", messages=[..]). Switching to Claude is a one-field change to the model string. The SDK handles streaming, async calls, token counting, and cost tracking per-request. Developers building with LangChain or LlamaIndex can drop LiteLLM in as the underlying model client to gain multi-provider routing without touching the framework layer above.

Where LiteLLM sits versus OpenRouter and Portkey

Three tools dominate LLM gateway conversations: LiteLLM, OpenRouter, and Portkey. They make fundamentally different architectural bets, and the right choice depends on your infrastructure philosophy rather than feature lists.

OpenRouter is a closed, hosted marketplace. You sign up, get one API key, and access 300-plus models including community fine-tunes through OpenRouter's servers. OpenRouter adds a 5.5% credit purchase fee and takes a revenue split with model providers who list on its marketplace. All your traffic flows through OpenRouter's infrastructure, which means zero deployment overhead but no control over data routing. For rapid prototyping and personal projects, OpenRouter is the fastest path to multi-model access. At $10,000 per month in LLM spend, you are paying OpenRouter $550 in fees. You cannot self-host it, audit its code, or deploy it in an air-gapped environment. It works best for developers who want model access, not infrastructure control.

Portkey is a closed, managed gateway that runs on Portkey's edge network. It prices per recorded log (around $49 per month for the Pro tier, covering 100K-3M requests), which means the cost scales with observability usage rather than token volume. Portkey's differentiator is built-in, production-grade observability: detailed request logs, latency tracing, cost attribution, semantic caching, and guardrail metrics are included in the managed service without requiring external integrations. Semantic caching, notably absent from LiteLLM's open-source tier, lets Portkey serve cached responses for semantically similar (not just identical) queries, which can meaningfully reduce costs at scale. Like OpenRouter, Portkey is not self-hostable and traffic flows through its infrastructure.

LiteLLM makes the opposite bet: you own the infrastructure and the data. The MIT license means you can audit the code, deploy it in regulated environments, and run it in air-gapped setups where traffic can never touch a third-party server. No data leaves your network except the direct calls to LLM providers you explicitly configure. The trade-off is operational burden. Running a production LiteLLM instance requires managing PostgreSQL, Redis, database migrations, backups, and connection pooling. Infrastructure costs for a realistic production setup run $2,000-$2,300 per month before factoring in DevOps labor. LiteLLM becomes the clear cost winner over OpenRouter once LLM spend exceeds roughly $10,000 per month, but only if your team has the capacity to maintain the stack.

"Switching providers is a one-line code change. That's the core promise, and for most providers it actually delivers." -- aicoolies.com reviewer, 2025

Developers using OpenRouter alongside LiteLLM often use OpenRouter for access to community and experimental models while routing production traffic through a self-hosted LiteLLM instance for compliance. The two are not mutually exclusive: LiteLLM can proxy to OpenRouter as one of its configured providers. For teams already using Helicone for observability, LiteLLM can coexist as the routing layer while Helicone handles the logging.

What the daily proxy reality looks like

Setup is fast. A working local instance runs in under ten minutes: install via pip, create a config YAML that lists your models and their provider credentials, run litellm --config config.yaml, and any OpenAI-compatible client hitting localhost:4000 is now routing through the proxy. Docker Compose setups add Postgres and Redis in another ten minutes for persistence and caching.

The YAML configuration is where teams spend most of their time. Model groups define fallback chains. Router settings control whether load balancing is round-robin, least-busy, or latency-weighted. Budget keys are issued per team with hard caps and soft alerts. For a platform team standardizing LLM access across a company, this config becomes the central policy document for all AI spend.

At moderate volume (under 100,000 requests per day), the proxy is largely transparent. Latency overhead for most calls is minimal. Fallback routing, budget enforcement, and provider-specific retry logic work without intervention.

At higher volume, the database becomes the choke point. LiteLLM writes request logs to PostgreSQL synchronously in the request path. Past a million accumulated log rows, the docs themselves acknowledge that API response times may degrade. Teams have reported cached requests with one-millisecond cache hits still returning 10-plus second end-to-end latencies due to database write serialization overhead. The operational workaround is disabling detailed logging or sharding the database, which defeats the purpose of the built-in spend tracking. This architectural constraint has pushed some high-throughput teams toward alternatives built on faster runtimes (Go-based proxies like Bifrost have emerged specifically to address this).

AI agents built with AnythingLLM or other agent frameworks can point to a LiteLLM proxy as their model backend, which gives platform teams centralized visibility over agent-originated LLM spend without requiring per-agent API key management.

Who LiteLLM is built for

LiteLLM works best for platform and backend engineering teams that manage LLM access for multiple internal squads, handle regulated data that cannot flow through third-party infrastructure, or are running enough LLM spend that the self-hosting cost is justified. Companies building AI products on multiple providers for resilience (Claude as primary, GPT-4o as fallback, Bedrock as the enterprise-required option) get the most out of the gateway's routing and fallback logic.

AI engineers prototyping across models use the Python SDK as a "try before you commit" layer: build against the litellm.completion() interface and switch providers without touching application logic. This is genuinely useful during the early stages of a product, when the right model for a task is still being evaluated.

Organizations in healthcare, finance, or other regulated industries appreciate the air-gapped deployment option and the ability to audit the full codebase for compliance. The self-hosted model means you can demonstrate to auditors that sensitive prompt data never touches a third-party server.

What LiteLLM is not

LiteLLM is not a good fit for individuals, solo developers, or small teams. The infrastructure overhead, the PostgreSQL and Redis dependencies, the YAML configuration model, and the operational maintenance burden are all calibrated for engineering teams with DevOps capacity. A developer who just wants to call Claude and GPT-4o from the same codebase is better served by the Python SDK alone (without the proxy) or by OpenRouter's zero-setup hosted option.

Skip LiteLLM if you need built-in semantic caching without additional integration work. OpenAI-format caching (exact-match) works; semantically similar query deduplication requires you to wire in an external layer yourself. For teams where observability is the primary requirement, Portkey's managed stack delivers more per dollar at low-to-moderate volume.

The March 2026 PyPI supply chain incident is relevant context for any team evaluating LiteLLM's Python package. Versions 1.82.7 and 1.82.8 were compromised by the "TeamPCP" threat actor group via a poisoned Trivy security scanner in BerriAI's CI/CD pipeline. The malicious packages, which harvested SSH keys, cloud credentials, and Kubernetes tokens, were live for approximately 40 minutes on March 24, 2026 before PyPI quarantined them. BerriAI's response was transparent: Krrish Dholakia and Ishaan Jaffer published a public timeline, engaged Google's Mandiant team for forensic analysis, rotated all credentials, and released clean versions through a hardened CI/CD pipeline. Customers running the official Docker image were not affected. Teams installing from PyPI directly should pin to v1.83.0 or later and verify the release pipeline before upgrading.

LiteLLM also does not host models. It is a routing and translation layer, not a compute provider. Teams running local inference through Ollama or vLLM can point LiteLLM at those servers, but LiteLLM itself has no GPU infrastructure.