Streaming

Parsing SSE, the usage chunk, cancellation, streaming errors, timeouts and keepalives on long requests, and when stream=true is required

The platform supports SSE (Server-Sent Events) streaming. The first token arrives sooner, so your UI can render as generation happens.

Basic usage#

Client setup (base_url and API key) is covered in the Quickstart; each snippet below repeats only the minimum.

OpenAI SDK#

from openai import OpenAI

client = OpenAI(
    base_url="https://qd.zsopc.com/v1",
    api_key="sk-gpushare-xxx",
)

stream = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Stream a haiku"}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)

Anthropic SDK#

from anthropic import Anthropic

client = Anthropic(
    base_url="https://qd.zsopc.com",  # ⚠️ no /v1
    api_key="sk-gpushare-xxx",
)

with client.messages.stream(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Stream a haiku"}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

Gemini SDK#

from google import genai

client = genai.Client(
    api_key="sk-gpushare-xxx",
    http_options={"base_url": "https://qd.zsopc.com", "api_version": "v1beta"},
)

stream = client.models.generate_content_stream(
    model="gemini-2.5-pro",
    contents="Stream a haiku",
)
for chunk in stream:
    print(chunk.text, end="", flush=True)

When stream=true is required#

ToolStreaming requiredWhy
image_generation✅the upstream WebSocket channel has no synchronous response — it only streams
web_search✅same, and the search progress comes back live as stream chunks
function toolseitherthey use the HTTP channel, streaming or not
Plain text chateitherstreaming or not

Otherwise the gateway returns 400 invalid_request_error with the message Tools `web_search` and `image_generation` require `stream: true`.

SSE wire format#

The gateway forwards the upstream SSE as-is: each data: line is one chunk.

OpenAI Chat stream chunks#

data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"}}]}

data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" world"}}]}

data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":10,"completion_tokens":2,"total_tokens":12}}

data: [DONE]

Two things hand-written parsers get wrong most often:

  • The finish_reason chunk carries no usage — usage arrives as a separate trailing chunk right after it, which is the standard stream_options.include_usage convention
  • The usage chunk's choices is an empty array, so chunk.choices[0].delta goes out of bounds — check chunk.choices is non-empty before reading delta

Anthropic stream events#

event: message_start
data: {"type":"message_start","message":{...}}

event: content_block_delta
data: {"type":"content_block_delta","delta":{"type":"text_delta","text":"Hello"}}

event: message_stop
data: {"type":"message_stop"}

Gemini streaming (SSE)#

data: {"candidates":[{"content":{"parts":[{"text":"Hello"}]}}]}

data: {"candidates":[{"content":{"parts":[{"text":" world"}]},"finishReason":"STOP"}]}

The Gemini protocol has no [DONE] sentinel — the stream closing is the end signal, so don't wait for [DONE].

Parsing by hand (curl, or a client with no SDK)#

curl https://qd.zsopc.com/v1/chat/completions \
  -H "Authorization: Bearer $PLATFORM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-6-luna",
    "stream": true,
    "messages": [{"role":"user","content":"Hello"}]
  }' \
  --no-buffer | while read line; do
    echo "$line"
  done

Parsing with JavaScript fetch:

const resp = await fetch("https://qd.zsopc.com/v1/chat/completions", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.PLATFORM_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "gpt-6-luna",
    stream: true,
    messages: [{ role: "user", content: "Hello" }],
  }),
});

const reader = resp.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });

  const lines = buffer.split("\n");
  buffer = lines.pop() ?? "";

  for (const line of lines) {
    if (!line.startsWith("data: ")) continue;
    const data = line.slice(6);
    if (data === "[DONE]") return;
    const chunk = JSON.parse(data);
    process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  }
}

Getting token usage#

OpenAI Chat streaming does not return a trailing usage chunk by default. To get real token counts, add stream_options:

stream = client.chat.completions.create(
    model="gpt-6-luna",
    messages=[...],
    stream=True,
    stream_options={"include_usage": True},  # the important bit
)

last_chunk = None
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")
    last_chunk = chunk

# the final chunk carries usage
print(f"\nTokens: {last_chunk.usage}")

Internally the gateway auto-injects stream_options.include_usage=true for most upstream channels (except GLM, which sends a usage chunk anyway). So you'll get token counts without setting it — but setting it explicitly is more robust.

Interrupting and cancelling#

OpenAI SDK (Python)#

stream = client.chat.completions.create(..., stream=True)
try:
    for chunk in stream:
        if some_user_canceled():
            stream.close()  # close explicitly; the gateway sees a client disconnect
            break
        print(chunk.choices[0].delta.content or "", end="")
except KeyboardInterrupt:
    stream.close()

fetch (JavaScript)#

const controller = new AbortController();

const resp = await fetch("https://qd.zsopc.com/v1/chat/completions", {
  signal: controller.signal,
  // ...
});

// then, to interrupt:
controller.abort();

When the client disconnects, the gateway's upstream connection closes immediately too (on both HTTP and WebSocket channels) — the mechanism is a dropped connection rather than an explicit cancel instruction, and the effect is that the upstream stops generating. Tokens already generated are still billed; if the disconnect means the trailing usage chunk never arrived, the gateway books output tokens estimated from the characters already streamed.

Error handling#

Before the stream opens: an ordinary HTTP error, not SSE#

Auth failure (401), exhausted balance (402), a nonexistent model (400 model_not_found) and no available channel (503) all happen before the stream opens and come back as an ordinary HTTP JSON error body — there's no SSE yet, so handle them like non-streaming errors. Full list: Error handling.

After the stream opens (HTTP 200): it depends on the path#

Tool requests (web_search / image_generation, on the WS V2 channel): a mid-stream error is synthesised into an error chunk followed by [DONE]:

data: {"error":{"message":"<the specific reason>","type":"api_error","code":"upstream_error"}}

data: [DONE]

type and code are fixed values (api_error / upstream_error), so don't branch on code to classify the error — the actual reason is only in the message text. When parsing, check for an error field before handling delta.content, and treat error as the end-of-turn signal.

Plain text chat and every /v1/messages, /v1/responses and /v1beta path: when the upstream fails mid-stream the gateway synthesises nothing and the stream simply ends early — there's no guarantee of [DONE], finish_reason, message_stop or any error frame. A robust client must treat "stream ended without a terminator" as an interruption, not a normal completion. (Error events native to the upstream protocol, such as Anthropic's event: error, are forwarded verbatim. The gateway also generates error events itself in the keepalive case described under Long requests and timeouts.)

On either path, whatever content streamed before the error or interruption is partially valid and can be kept on screen.

Long requests and timeouts#

There is no 180-second total ceiling. The limits on one upstream call are:

CaseLimit
Non-streaming request600 seconds
Streaming request (stream: true), whole streamabout 61 minutes (3660 seconds)
Streaming request, waiting for the upstream to start responding120 seconds (not applied on the Gemini streaming endpoint)

When the upstream sends nothing at all for 300 seconds in the middle of a stream, most channels treat it as gone and end the stream. Retriable failures switch channels automatically, but only within the first 60 seconds of the request, and one request tries at most 3 channels.

So that proxies along the way don't drop a connection that has carried no data for a while, the gateway answers long requests with HTTP 200 first and then keeps the connection alive:

  • Streaming (/v1/chat/completions, /v1/messages, /v1/responses): if the upstream hasn't started responding 20 seconds after the request arrived, the gateway sends 200 with text/event-stream, then an SSE comment line : keepalive every 15 seconds. Standard SSE parsers ignore comment lines.
  • Non-streaming (those three endpoints and /v1beta/...:generateContent): if the call hasn't finished after 20 seconds, the gateway sends 200, writes one space every 15 seconds, and then the full JSON body (JSON allows leading whitespace). These responses carry the header x-gpushare-nonstream-keepalive: 1.

The cost: a failure after the 20-second mark can't return a 5xx status. A streaming request gets an in-stream error event instead (Chat: a data: line with error followed by [DONE]; Responses: response.failed; Anthropic: event: error). A non-streaming request gets HTTP 200 with the error in the body's error field. Decide success from the body, not from the status code alone. The Gemini streaming endpoint (:streamGenerateContent) has no keepalive before the stream opens.

What to do:

  • Use streaming for long outputs; tokens already produced are billed as usual
  • If your client enforces a deadline on the whole request, give non-streaming requests at least 660 seconds (the 600-second limit plus room for a channel switch). Streaming time grows with the length of the output, so time out on "no data received for a while" rather than on a fixed total

Per-endpoint timeouts: API reference.

Reverse proxies and Nginx#

The gateway sets these response headers:

Content-Type: text/event-stream
Cache-Control: no-cache, no-store, no-transform
X-Accel-Buffering: no

If you proxy this gateway behind your own reverse proxy, make sure buffering is off (proxy_buffering off; in Nginx) — otherwise the whole streamed response is buffered until it completes and delivered in one go, which defeats the point.

Debugging#

Watching the raw SSE with curl is the most direct approach:

curl https://qd.zsopc.com/v1/chat/completions \
  -H "Authorization: Bearer $PLATFORM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-6-luna","stream":true,"messages":[{"role":"user","content":"hi"}]}' \
  --no-buffer 2>&1 | head -20

If you see the response arrive all at once instead of chunk by chunk, something in between is buffering.