What we do
When an upstream provider degrades, our job is to be transparent about it quickly rather than to paper over it.
- Retry idempotent failures. A connection reset or an upstream 5xx before any tokens have streamed is retried once against a different upstream endpoint, transparently.
- Fail fast once streaming has started. If tokens are already flowing, we surface the error rather than silently restarting, because a restarted stream would corrupt your output.
- Return
503 lcllm_upstream_unavailablewith aretry-afterwhen the provider is genuinely down, so your backoff has something honest to work with. - Post to the status page within minutes, with the affected provider and models named.
What we deliberately don’t do
We never fail over to a different model. Not to a cheaper one, not to a comparable one, not to a different provider. If you asked for a specific model and it is unavailable, you get an error.
This is the single most important guarantee we make. Cross-model failover would mean your production traffic silently changing behaviour during an incident, with different capabilities, different formatting, and different results, at exactly the moment you are least able to notice. A model swap you did not ask for is a correctness bug, not a resilience feature.
What to build on your side
Because we will not choose a fallback model for you, choose it yourself: explicitly, where you can see it and test it.
- Catch
503and decide in your own code whether a different model is acceptable for that particular call. For a summarisation job it usually is; for a tool-calling agent it usually is not. - Keep a direct provider key as a break-glass path. If we are the thing that is down, one environment variable gets you back to the provider at list price, which is worse than our price and infinitely better than being offline.
- Subscribe to status notifications so a human on your side knows before your users tell them.