Keep Your Agent Running When the Primary Model Is Down: LiteLLM Router Fallback in 10 Lines

TL;DR: LiteLLM Router's fallback_models list reroutes failed API calls to a second provider on any 5xx, 429, or timeout — the entire failover chain lives in your router config, not your application code.

Today's Claude incident (the eighth this month) makes single-provider agent architecture a reliability decision, not a cost-saving default. LiteLLM Router lets you add a fallover chain without touching application logic:

import os
from litellm import Router

router = Router(
    model_list=[
        {"model_name": "primary", "litellm_params": {
            "model": "anthropic/claude-opus-5-20260801",
            "api_key": os.environ["ANTHROPIC_API_KEY"]
        }},
        {"model_name": "fallback-1", "litellm_params": {
            "model": "openai/gpt-5-mini",
            "api_key": os.environ["OPENAI_API_KEY"]
        }},
    ],
    fallback_models=["fallback-1"],
    num_retries=2,
    retry_after=1,   # base delay in seconds (LiteLLM adds jitter)
    allowed_fails=3,
)

response = router.completion(model="primary", messages=messages)

What triggers fallover: 5xx errors, 429 rate-limits, context-window overflow, content-policy blocks, and timeouts. The router retries the primary num_retries times before stepping to the next entry in fallback_models.

Three limits to know before you ship

1. Output drift. Fallback models produce structurally different responses. If your primary returns clean JSON and the fallback is more verbose, your downstream parser breaks silently. Test the fallback path in staging before production fires it.
2. Cost spikes. If your primary is the cheapest option and the fallback is 5× the price, a three-hour outage during peak traffic generates a significant unexpected bill. Set a cost ceiling or alert if the fallback is active for more than a few minutes.
3. Jitter is not optional. Retries without jitter create a thundering herd against an already-stressed provider. LiteLLM adds jitter by default on retry_after — confirm you have not overridden it with a fixed delay.

Sources: Fallbacks (Provider Failover) — LiteLLM docs · LLM API Resilience in Production: Rate Limits, Failover, and Naive Retry Costs — TianPan.co · LLM Gateway in Production: Multi-Provider Routing with LiteLLM — DevOpsBoys