Litelm: LiteLLM Without the Bloat

Sep 12, 2026 01:10 AM - 2 days ago 7

PyPI Python MIT

litellm's routing + translator successful ~2,900 lines and 2 limitations (openai, httpx).

litellm routes LLM calls crossed providers and translates betwixt connection formats. That halfway is buried nether 100k+ LOC of proxy servers, caching layers, costs tracking, and dozens of features astir users ne'er touch. litelm extracts conscionable the telephone way — exemplary routing, connection translation, streaming, instrumentality use, embeddings — and thing else. No Router class, nary proxy, nary caching.

pip instal litelm # openai + httpx pip instal litelm[anthropic] # + anthropic SDK pip instal litelm[bedrock] # + boto3 pip instal litelm[all] # everything

import litelm # Basic completion response = litelm.completion("openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}]) print(response.choices[0].message.content) # Streaming for chunk in litelm.completion("groq/llama-3.1-70b-versatile", messages=[...], stream=True): print(chunk.choices[0].delta.content or "", end="") # Embeddings response = litelm.embedding("openai/text-embedding-3-small", input=["hello world"])

Every usability has an async variant: acompletion, aembedding, aresponses, atext_completion.

The API mirrors litellm — aforesaid usability names, aforesaid arguments, aforesaid consequence types. If you're utilizing litellm today, switching is s/litellm/litelm/ successful your imports.

litellm litelm

Model routing (provider/model → correct endpoint)	✓	✓
Message translator (Anthropic, Bedrock, Cloudflare, Mistral)	✓	✓
Streaming + stream_chunk_builder	✓	✓
Tool usage (function calling)	✓	✓
Embeddings	✓	✓
Text completions	✓	✓
OpenAI Responses API	✓	✓
Mock responses	✓	✓
Router (load balancing, fallbacks)	✓	✗
Proxy server	✓	✗
Caching / budgeting / costs tracking	✓	✗
Token counting	✓	✗
Image gen, audio, OCR, fine-tuning	✓	✗
Agents, guardrails, scheduler	✓	✗

Routes to 19 providers via "provider/model-name" syntax. Any OpenAI-compatible endpoint useful via api_base.

Provider Env Var Handler Verified

OpenAI	OPENAI_API_KEY	OpenAI SDK	Yes
Anthropic	ANTHROPIC_API_KEY	Custom	Yes
Groq	GROQ_API_KEY	OpenAI-compat	Yes
Mistral	MISTRAL_API_KEY	Custom	Yes
xAI	XAI_API_KEY	OpenAI-compat	Yes
OpenRouter	OPENROUTER_API_KEY	OpenAI-compat	Yes
Azure	AZURE_API_KEY	OpenAI SDK (Azure)	Yes
Bedrock	AWS_ACCESS_KEY_ID	Custom	No
Cloudflare	CLOUDFLARE_API_TOKEN	Custom	No
Together	TOGETHERAI_API_KEY	OpenAI-compat	No
Fireworks	FIREWORKS_API_KEY	OpenAI-compat	No
DeepSeek	DEEPSEEK_API_KEY	OpenAI-compat	No
Perplexity	PERPLEXITYAI_API_KEY	OpenAI-compat	No
DeepInfra	DEEPINFRA_API_TOKEN	OpenAI-compat	No
Gemini	GEMINI_API_KEY	OpenAI-compat	No
Cohere	COHERE_API_KEY	OpenAI-compat	No
Ollama	—	OpenAI-compat	No
vLLM	—	OpenAI-compat	No
LM Studio	—	OpenAI-compat	No

Set the situation adaptable for your provider:

export OPENAI_API_KEY=sk-... export ANTHROPIC_API_KEY=sk-ant-...

Or walk directly:

litelm.completion("openai/gpt-4o", messages=[...], api_key="sk-...") litelm.completion("openai/gpt-4o", messages=[...], api_base="http://localhost:8000/v1")

All supplier errors are mapped to litelm's objection hierarchy:

from litelm import ContextWindowExceededError, RateLimitError, AuthenticationError try: response = litelm.completion("openai/gpt-4o", messages=messages) except ContextWindowExceededError: # punctual excessively agelong — truncate and retry pass except RateLimitError: # backmost off pass except AuthenticationError: # bad API key pass

tools = [{"type": "function", "function": { "name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}, }}] response = litelm.completion( "openai/gpt-4o", messages=[{"role": "user", "content": "Weather successful Paris?"}], tools=tools, tool_choice="required", ) tool_call = response.choices[0].message.tool_calls[0] print(tool_call.function.name, tool_call.function.arguments)

Any OpenAI-compatible server useful via api_base:

# vLLM litelm.completion("openai/my-model", messages=[...], api_base="http://localhost:8000/v1") # Ollama litelm.completion("ollama/llama3", messages=[...], api_base="http://localhost:11434/v1") # LM Studio litelm.completion("openai/local-model", messages=[...], api_base="http://localhost:1234/v1")

litelm is human-directed, AI-assisted software. Much of the codification was written pinch Claude Code utilizing Claude Opus 4.6/4.7. Code written from 2026-05-14 onward is written done Pi utilizing GPT-5.5. Compatibility claims are based connected tests and maintainer review, not AI authorship.

Maintainer attestation, 2026-09-11: LiteLLM's routing/formatting changes were reviewed from 649eb2d done 9a715df2. The audit triaged 360 core-path commits, inspected upstream tests for perchance applicable behavior, and fixed the resulting compatibility gaps test-first. Local scoped tests: 262 passed, 55 skipped; each 45 available-provider unrecorded tests and each 10 DSPy fume tests besides passed pinch the existent dependency lock.

This attests litelm's declared routing/formatting/DSPy aboveground only, not afloat litellm compatibility.

Alpha. 262 ain tests passing. The existent scoped LiteLLM 9a715df2 baseline has 75 passing ported tests and nary remaining actionable assertion/runtime failures.

DSPy drop-in verified — each 7 execution paths proven unrecorded (Predict, CoT, typed signatures, streaming, embeddings, instrumentality use, multi-output).

uv tally --extra each pytest tests/ -x --ignore=tests/ported --timeout=10 # 262 non-live tests bash scripts/ported_contract.sh # 49 accelerated upstream statement tests uv tally --extra each pytest tests/test_live.py -m unrecorded --timeout=30 # 45 unrecorded supplier tests uv tally pytest tests/test_dspy_smoke.py -m unrecorded --timeout=60 # 10 DSPy integration tests

Live tests require API keys successful .env.test. Skipped by default; tally pinch -m live.