AI API Streaming with SSE: Formats, Nginx Buffering, Pitfalls

Turning on streaming is one line: "stream": true. Making it work in production is a different job. The classic failure reports look like this: it streams token by token on localhost, but in production the page sits blank for fifteen seconds and then dumps the whole answer at once. Or the output stops halfway and the UI shows it as finished. Or non-ASCII text occasionally turns into replacement characters. Or a user clicks stop and tokens keep being billed.

All of these come from the layers between the model and the user, not from the model. This guide walks through the full path: what Server-Sent Events look like on the wire, how OpenAI and Anthropic formats differ, why nginx holds your stream hostage, how to read a stream correctly with fetch, how to detect and recover from drops, and how streamed requests are billed. For SDK-level setup, start with the Node.js quickstart.

SSE in One Paragraph

Server-Sent Events is an ordinary HTTP response that does not end quickly. The server responds with Content-Type: text/event-stream and keeps writing lines. Lines starting with data: carry payload, an optional event: line names the event type, lines starting with a colon are comments (often used as heartbeats), and a blank line terminates each event. It is one-directional and plain HTTP, which is exactly what "send one request, receive a long answer" needs, and it passes through proxies far more easily than WebSockets.

The quickest way to see it is curl with output buffering disabled:

curl -N https://zivv.pro/v1/chat/completions \
  -H "Authorization: Bearer sk-your-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-5",
    "stream": true,
    "messages": [{"role": "user", "content": "Count to five"}]
  }'

Keep this command handy. It is your baseline: if direct curl streams smoothly and your app does not, the problem lives in your stack.

OpenAI vs Anthropic Event Formats

OpenAI-compatible streams (/v1/chat/completions) use only data: lines. Each carries a chat.completion.chunk, and the stream ends with a literal data: [DONE]:

data: {"choices":[{"index":0,"delta":{"content":"One, two"},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Anthropic-native streams (/v1/messages) use named events that follow the message lifecycle:

event: message_start
data: {"type":"message_start","message":{"usage":{"input_tokens":12,"output_tokens":1}}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"One, two"}}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":15}}

event: message_stop
data: {"type":"message_stop"}

The /v1/responses API used by Codex has its own event names, such as response.output_text.delta and response.completed, and Gemini streaming has yet another shape. The practical rule: use the official SDKs whenever you can, because they already normalize all of this. If you must parse by hand, write one parser per protocol. Also expect non-text deltas in Anthropic streams, such as thinking_delta or input_json_delta for tool use; a text-only parser should skip them, not crash.

Why Nginx Makes Streams Freeze

"Works locally, arrives all at once in production" is almost always response buffering. Nginx enables proxy_buffering by default: it collects the upstream response and forwards it when buffers fill or the response ends. Great for web pages, fatal for SSE. Turn it off where you proxy model traffic:

location /api/chat {
    proxy_pass http://127.0.0.1:3000;
    proxy_http_version 1.1;
    proxy_set_header Connection "";

    proxy_buffering off;      # do not hold the upstream response
    proxy_cache off;          # never cache streams
    gzip off;                 # compression also accumulates data

    proxy_read_timeout 300s;  # max gap between two reads
    proxy_send_timeout 300s;
}

If you cannot edit nginx, have the application send X-Accel-Buffering: no, which disables buffering for that response only. Then check the other usual suspects:

• Compression middleware in your web framework that waits for enough bytes to compress
• CDNs and WAFs that buffer or cache responses unless streaming paths are excluded
• Serverless platforms that return a function's response only once it completes unless you use their streaming response mode
• Your own handler, if it reads the entire upstream stream before writing anything

Note that proxy_read_timeout measures the gap between reads, not total duration. Models can pause before output while they reason, and nginx's 60-second default will cut long-thinking requests mid-flight. The 529 and timeout guide covers timeout layering in detail.

Reading a Stream in the Browser

The browser's EventSource only does GET and cannot send a body or auth header, so it is the wrong tool for model APIs. Use fetch with POST and read response.body yourself:

async function streamChat(messages: unknown[], onText: (t: string) => void, signal?: AbortSignal) {
  const res = await fetch("/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ messages, stream: true }),
    signal,
  });
  if (!res.ok || !res.body) throw new Error("HTTP " + res.status);

  const reader = res.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";
  let finished = false;

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });
    const events = buffer.split("\n\n");
    buffer = events.pop() ?? ""; // keep the incomplete tail

    for (const evt of events) {
      for (const line of evt.split("\n")) {
        if (!line.startsWith("data:")) continue;
        const data = line.slice(5).trim();
        if (data === "[DONE]") { finished = true; continue; }
        const text = JSON.parse(data).choices?.[0]?.delta?.content;
        if (text) onText(text);
      }
    }
  }
  if (!finished) throw new Error("stream ended without [DONE]");
}

Three bugs this avoids:

1. Split events. Network chunk boundaries have nothing to do with event boundaries; a JSON payload can arrive in two pieces. Buffer, split on blank lines, and carry the tail forward.
2. Broken characters. Without { stream: true }, a multi-byte character split across chunks decodes as garbage.
3. False completion. done only means the connection closed. Only [DONE] (or message_stop on Anthropic) means the answer is complete.

The /api/chat endpoint here should be your own server route that holds the key and calls https://zivv.pro/v1. Never ship a key to the browser.

Stop Buttons and Cancellation

Wire stop to an AbortController and pass its signal to fetch. Then make sure your server propagates the disconnect upstream, for example by passing the incoming request's signal into the SDK call. Otherwise the browser stops reading while your server keeps consuming, and paying for, tokens nobody will see.

Detecting and Recovering from Drops

• Trust end markers, not connection close. No [DONE] or message_stop means the stream was cut.
• Before the first byte, retry automatically. Nothing reached the user, so a retry is invisible.
• After partial output, do not silently retry. It duplicates text. Keep what arrived and offer continue or regenerate.
• Continue instead of restarting for long outputs: send the partial text back as an assistant message and ask the model to resume, which costs fewer tokens than starting over.
• Watch for in-stream errors. Anthropic can send event: error after a 200 status, so status codes alone are not enough.

There is no Last-Event-ID style resume for model streams; reconnecting means a new request. For backoff, reuse the exponential retry from the 429 guide.

How Streaming Is Billed

Streaming does not change the price. Requests are billed on actual input and output tokens at the group multiplier times the official model price, whether streamed or not; see billing overview. Tokens generated before a stop or disconnect are generally still counted, which is why prompt cancellation matters. To capture usage while streaming, send "stream_options": {"include_usage": true} on OpenAI-format requests and read the final chunk, whose choices array is empty, so guard your parser. On Anthropic streams, read message_start and message_delta. Treat your own counters as live estimates and the console usage records as the source of truth.

Pre-Launch Checklist

FAQ

Why does my AI stream work locally but arrive all at once in production? Buffering in a middle layer: nginx proxy_buffering, compression middleware, a CDN or a serverless runtime. Disable it as shown above or send X-Accel-Buffering: no, then verify layer by layer with curl -N.

Can I use EventSource for OpenAI or Claude streaming? No. It cannot send POST bodies or custom auth headers. Use fetch and parse response.body.

How do I know if a stream finished or was cut off? Check for the protocol's end marker: [DONE] and a finish_reason for OpenAI format, message_stop for Anthropic.

Is streaming more expensive than non-streaming? No. Billing is per actual token either way.

Do Claude Code or Cursor need streaming configured? No, they stream by default; you only set the base URL and key, as in the Cursor setup guide. If you put your own proxy in front of them, disable buffering there too.

Streaming problems are pipeline problems. Get every hop passing bytes through immediately, enforce end markers, and propagate cancellation, and the "freezing" and "half answers" disappear. Create a Zivv account, run the curl command above, and browse models in the Model Hub.