Fix 529 Overloaded Errors, API Timeouts, and Dropped Streams

A 529 with overloaded_error in the body is the upstream provider saying its capacity is saturated right now. There is nothing wrong with your request, your key, or your balance, and no configuration change on your side will make an overloaded cluster less overloaded. The related failures, requests that time out before returning anything and streams that die mid-response, get lumped in with 529 because they feel the same, but each one needs a different fix.

Tell the Three Failures Apart

Getting the diagnosis right matters because the fixes conflict: retrying is correct for 529, while retrying a timed-out 300-second generation with the same 30-second client timeout just fails the same way, slower.

Retry 529 Correctly

529 is transient by definition, so it is one of the safest statuses to retry, with two caveats: overload windows last longer than rate-limit windows, and hammering an overloaded upstream at full concurrency extends the outage for everyone, including you.

• Use exponential backoff with jitter: roughly 1s, 2s, 4s, 8s, plus a random fraction
• Drop your concurrency while retrying; do not resend a 50-request batch the moment one probe succeeds
• Cap attempts at 3 or 4, then fall back instead of waiting indefinitely

The backoff pattern is the same as for 429, and the 429 guide has a full implementation you can reuse; only the trigger status differs.

Set Timeouts for Generation, Not for Web Pages

Default HTTP client timeouts assume you are fetching a web page in a few seconds. A large-max_tokens completion on a heavyweight model can legitimately run for minutes, and a 30-second default will kill healthy requests and report them as failures. Split the timeout into parts:

import httpx

client = httpx.Client(
    timeout=httpx.Timeout(
        connect=10.0,   # reaching the server should be fast
        read=300.0,     # generation can legitimately take minutes
        write=30.0,
        pool=10.0,
    )
)

The connect timeout should stay short, because failing to reach the server in ten seconds is a real problem you want surfaced quickly. The read timeout is the one that must be generous. With streaming enabled there is a bonus: most clients apply the read timeout between chunks rather than across the whole response, so a model that streams steadily for five minutes never trips a 300-second read timeout, while a genuinely hung connection still fails fast.

Handle Dropped Streams

A healthy streamed response ends with an explicit termination signal: a finish reason in the final chunk, or a [DONE] event on OpenAI-style SSE. If the iterator simply stops without one, you received a partial answer, and silently using it is worse than failing. Detect it:

finish_reason = None
collected = []

for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        collected.append(delta.content)
    if chunk.choices[0].finish_reason is not None:
        finish_reason = chunk.choices[0].finish_reason

if finish_reason is None:
    # The stream died mid-response: treat the output as incomplete
    handle_dropped_stream("".join(collected))

Recovery options, in order of preference:

1. Re-run the whole request when the output is cheap and the request is idempotent. Simplest and most reliable.
2. Continue from the partial output for long generations: send a follow-up request with the truncated text and an instruction to continue exactly where it stopped, then stitch the parts. Verify the seam.
3. Checkpoint upstream so a retry never repeats side effects: write results only after finish_reason confirms completion.

Split Long Tasks

The longer a single request runs, the more overload windows and connection resets it is exposed to. A 20-minute monolithic generation is a bet that nothing goes wrong for 20 minutes; five 4-minute chunks each risk only their own window and give you resume points for free.

• Cap max_tokens per call and continue across calls instead of requesting one enormous output
• Chunk long documents and merge per-chunk results
• In agent workflows, separate the planning call from execution calls so a failure costs one step, not the plan
• In Claude Code sessions, commit between logical steps so an interrupted step is a small retry, not a lost session

Route Around Sustained Overload

Backoff handles a bad minute. A bad afternoon on a single provider needs a different answer: send the work somewhere else. If your stack is hardwired to one provider, that is an architectural decision to revisit. Behind a gateway, falling back is a one-line model change on the same endpoint and key: shift from claude-opus-4-8 to claude-sonnet-5, or across providers to gpt-5.5, and keep the pipeline moving. Zivv exposes 100+ models across all three protocols behind one key, current IDs are in Model Hub, and the error reference documents how upstream errors are surfaced.

FAQ

Am I billed for requests that end in 529 or time out? A 529 returns no completion, so there is nothing to bill. For dropped streams, tokens already generated may be counted depending on the platform; your usage page is the source of truth, and checking it is also how you confirm whether a timed-out request ever reached the backend.

Does a 529 mean I sent too many requests? No. That is a 429, which is about your usage. A 529 is about the provider's aggregate load and can hit you on your very first request of the day.

Does streaming prevent timeouts and drops? It prevents most read-timeout failures on long generations, because data flows continuously. It does not prevent connection drops, which is why the incomplete-stream detection above should be standard in anything long-running.

How many retries before falling back to another model? Three or four attempts over roughly 15 to 30 seconds. If a provider is still overloaded after that, the window is long, and a fallback model that answers now beats a perfect model that answers eventually.

Overload and timeouts are weather: you do not fix them, you build for them. Generous read timeouts, honest stream handling, chunked tasks, and a fallback route cover nearly every case. If your coding sessions are the workload being interrupted, the vibe coding setup shows the multi-model arrangement that keeps them running.