One Endpoint, 100 Providers: Run a Self-Hosted AI Gateway With LiteLLM in 5 Minutes

TL;DR: LiteLLM is a self-hosted OpenAI-compatible proxy that routes to 100+ LLM providers — Claude, Gemini, local Ollama, and more — behind a single endpoint, swappable without changing application code.

What you'll be able to do after this:

• Point any OpenAI SDK client at your own gateway and switch between providers by editing one config line
• Assign per-team virtual API keys with spend caps and automatic model fallbacks
• Add response caching, latency-based routing, and centralized logging in a single config.yaml

Install and call any model (SDK mode)

pip install litellm

from litellm import completion

# Same call format for every provider — just change the model string
r = completion(model="anthropic/claude-sonnet-4-6",
               messages=[{"role": "user", "content": "Hello"}])

r = completion(model="gemini/gemini-2.0-flash",
               messages=[{"role": "user", "content": "Hello"}])

r = completion(model="ollama/llama3.2",        # local, no API key
               messages=[{"role": "user", "content": "Hello"}])

Run the proxy gateway (team mode)

config.yaml:

model_list:
  - model_name: fast          # alias your app uses
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: fast          # second option for "fast" — LiteLLM load-balances
    litellm_params:
      model: ollama/llama3.2
      api_base: http://localhost:11434
  - model_name: smart
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  routing_strategy: latency-based-routing
  fallbacks: [{"fast": ["smart"]}]   # fall back to smart if fast is down

general_settings:
  master_key: sk-my-gateway-key      # issue virtual keys from this

litellm --config config.yaml --port 4000

Now any OpenAI client works unchanged:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:4000", api_key="sk-my-gateway-key")
resp = client.chat.completions.create(model="smart", messages=[...])

What to add next

• Virtual keys: POST /key/generate with max_budget + team_id — each team gets their own key with spend tracking.
• Caching: litellm_settings: cache: true — identical requests return cached responses instantly.
• Logging: set success_callback: ["langfuse"] to wire tracing without touching app code.

Security note: LiteLLM had a CVSS 10.0 RCE chain patched in v1.43.x (CVE-2026-42271). Always pin to the latest stable release before deploying.

Sources: LiteLLM Getting Started — docs.litellm.ai · LiteLLM Proxy (AI Gateway) — docs.litellm.ai · BerriAI/litellm — GitHub