← Back to Blog

AI API High Availability: Model Fallback and Cost Lookup

Zivv••18 min read
high availabilitymodel fallbackbest practices

Once a production feature depends on a language model, the scary failure is not a mediocre answer — it is no answer at all. Upstream rate limits, peak-hour overload or a model being retired all surface to your users as errors. The usual fix is retry and degrade logic in application code, but retrying the same model often fails again, and hand-rolled fallbacks have plenty of edge cases. Zivv offers request-level model fallback at the gateway: when the primary model is unavailable, the request moves to a backup model automatically, in a single round trip. This article explains how to use it, exactly when it switches, and how to find out which model served each request and what it cost.

Two Layers of Protection: Channels First, Then Models

Every request through Zivv has two layers of fault tolerance:

  1. Channel layer: a model is usually served by several upstream channels. When one is rate-limited, overloaded or failing, the gateway retries or moves to another channel for the same model, depending on the error. No configuration needed
  2. Model layer: if every channel for the model is unavailable, a request that lists backup models continues on the next one. That is the fallback covered here

Most transient failures are absorbed at the channel layer. The model layer handles bigger problems: a model that is rate-limited or overloaded across the board, or one that has been retired.

Configuration: Add models to the Request Body

Add a models array at the top level of the request body, listing backups in priority order (the field name matches OpenRouter's):

curl https://zivv.pro/v1/chat/completions \
  -H "Authorization: Bearer sk-your-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-5-5",
    "models": ["claude-opus-5-5", "claude-haiku-5-5"],
    "messages": [{"role": "user", "content": "Hello"}]
  }'

The rules:

  • Supported on /v1/chat/completions, /v1/messages and /v1/responses, including the Responses WebSocket mode
  • Up to three backups are used; duplicates and entries equal to model are dropped
  • The models field is stripped before forwarding upstream; a malformed value (anything other than an array of strings) returns 400
  • After switching, key allowlists, pricing, content moderation, routing and billing all run again for the new model

When the Gateway Switches Models

It only switches when another model might actually succeed:

SituationSwitches?
The model has no available channelYes
Every channel is rate-limited, overloaded or failing (429, 5xx, 529)Yes
The model was retired, has no pricing, or is not on the key's model allowlistYes
The request itself is invalid (a deterministic 4xx such as a bad parameter)No
The content was blockedNo
The key's group lacks permissionNo
The client disconnectedNo
Output has already started streaming to the clientNo

The last row matters for streaming: once the first chunk reaches you, the model will not change mid-answer, so you never get a response stitched together from two models. Invalid requests are not retried on another model for a similar reason — a bad parameter fails on every model, and switching would only waste time.

Using It from Code

With the OpenAI Python SDK, pass models via extra_body and use with_raw_response to read the headers:

from openai import OpenAI

client = OpenAI(base_url="https://zivv.pro/v1", api_key="sk-your-key-here")

raw = client.chat.completions.with_raw_response.create(
    model="claude-sonnet-5-5",
    messages=[{"role": "user", "content": "Summarize the key points of this weekly report."}],
    extra_body={"models": ["claude-opus-5-5", "claude-haiku-5-5"]},
)
resp = raw.parse()
print("served by:", raw.headers.get("x-zivv-model"))
print("request id:", raw.headers.get("x-request-id"))
print(resp.choices[0].message.content)

The Anthropic SDK supports extra_body too:

import anthropic

client = anthropic.Anthropic(base_url="https://zivv.pro", api_key="sk-your-key-here")

msg = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
    extra_body={"models": ["claude-opus-5-5"]},
)

The X-Zivv-Model response header names the model that served the request. It can be missing in one case: when the gateway had already sent headers as a keepalive during an earlier attempt. Then rely on /v1/generation, described below.

Choosing Backup Models

  • Comparable capability: a backup must be able to do the job. Use the same tier or one above for normal work; dropping a tier is fine when availability matters most
  • Different vendors are more resilient: models from the same vendor can fail together, so include one from another provider. See the Claude model comparison and the DeepSeek, Qwen and GLM guide
  • The key must have access: backups have to be in the key's groups; use a smart routing key with several groups when they span groups. Missing group permission does not trigger a switch
  • Protocol and feature support: for /v1/messages requests, backups must support the Anthropic protocol; if you rely on tool use or structured output, make sure the backups support them
  • Prompt compatibility: prompts tuned for one model may perform worse on another, so test your prompts against each backup before going live

Per-Request Lookup: /v1/generation

Every response carries an X-Request-ID header. Use it to look up the full story of that request:

curl "https://zivv.pro/v1/generation?id=<X-Request-ID>" \
  -H "Authorization: Bearer sk-your-key-here"

The data object includes:

  • model / upstream_model: the model that finally served the request
  • status_code, duration_ms, ttft_ms: final status, total duration and time to first token
  • total_cost: the cost summed over every attempt, plus input_tokens, output_tokens, cached_tokens and other token counts
  • models_tried: the models attempted, in order
  • attempts: per-attempt details

A few notes: you can only look up your own requests — anything else, including ids that do not exist, returns 404, and no channel information is ever included. Usage is written in batches after a request finishes, so a request that just completed may take a few seconds to appear. Team keys are attributed to the member who made the call.

This endpoint fits nicely into your own monitoring: sample requests periodically, track how often fallback kicks in and what it costs, and if the primary model keeps getting replaced, change the default.

FAQ

Q: Can Claude Code use model fallback? A: Claude Code does not let you add custom fields to the request body, so it cannot send models. Channel-level switching still applies to Claude Code, though, and that absorbs most everyday rate limiting and overload.

Q: How is this different from SDK retries? A: An SDK retry resends the same request to the same model. Fallback switches to a different model inside the gateway, in one round trip, with no degrade logic in your client. You can use both at once.

Q: Am I charged for failed attempts? A: Go by total_cost from /v1/generation: it adds up every attempt for the request, and it matches the usage records for that request ID in the console.

Q: How many backup models can I list? A: Up to three take effect; anything beyond the first three is ignored. If you need more than that, the default model is probably the wrong choice.

Wrapping Up

Gateway-level fallback turns switching models into a one-line change: add models, and when the primary has trouble the request lands on a backup, with X-Zivv-Model and /v1/generation showing exactly what happened. Combined with the multi-model routing guide and the 429 troubleshooting guide, it makes AI features in production noticeably more resilient. Sign up for Zivv — fallback works out of the box, with nothing extra to enable.