Streaming
Parsing SSE, the usage chunk, cancellation, streaming errors, timeouts and keepalives on long requests, and when stream=true is required
The platform supports SSE (Server-Sent Events) streaming. The first token arrives sooner, so your UI can render as generation happens.
Basic usage#
Client setup (base_url and API key) is covered in the Quickstart; each snippet below repeats only the minimum.
OpenAI SDK#
from openai import OpenAI
client = OpenAI(
base_url="https://qd.zsopc.com/v1",
api_key="sk-gpushare-xxx",
)
stream = client.chat.completions.create(
model="claude-sonnet-4-6",
messages=[{"role": "user", "content": "Stream a haiku"}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content or ""
print(delta, end="", flush=True)
Anthropic SDK#
from anthropic import Anthropic
client = Anthropic(
base_url="https://qd.zsopc.com", # ⚠️ no /v1
api_key="sk-gpushare-xxx",
)
with client.messages.stream(
model="claude-sonnet-4-6",
max_tokens=1024,
messages=[{"role": "user", "content": "Stream a haiku"}],
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
Gemini SDK#
from google import genai
client = genai.Client(
api_key="sk-gpushare-xxx",
http_options={"base_url": "https://qd.zsopc.com", "api_version": "v1beta"},
)
stream = client.models.generate_content_stream(
model="gemini-2.5-pro",
contents="Stream a haiku",
)
for chunk in stream:
print(chunk.text, end="", flush=True)
When stream=true is required#
| Tool | Streaming required | Why |
|---|---|---|
image_generation | ✅ | the upstream WebSocket channel has no synchronous response — it only streams |
web_search | ✅ | same, and the search progress comes back live as stream chunks |
function tools | either | they use the HTTP channel, streaming or not |
| Plain text chat | either | streaming or not |
Otherwise the gateway returns 400 invalid_request_error with the message Tools `web_search` and `image_generation` require `stream: true`.
SSE wire format#
The gateway forwards the upstream SSE as-is: each data: line is one chunk.
OpenAI Chat stream chunks#
data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"}}]}
data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" world"}}]}
data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":10,"completion_tokens":2,"total_tokens":12}}
data: [DONE]
Two things hand-written parsers get wrong most often:
- The
finish_reasonchunk carries no usage — usage arrives as a separate trailing chunk right after it, which is the standardstream_options.include_usageconvention - The usage chunk's
choicesis an empty array, sochunk.choices[0].deltagoes out of bounds — checkchunk.choicesis non-empty before readingdelta
Anthropic stream events#
event: message_start
data: {"type":"message_start","message":{...}}
event: content_block_delta
data: {"type":"content_block_delta","delta":{"type":"text_delta","text":"Hello"}}
event: message_stop
data: {"type":"message_stop"}
Gemini streaming (SSE)#
data: {"candidates":[{"content":{"parts":[{"text":"Hello"}]}}]}
data: {"candidates":[{"content":{"parts":[{"text":" world"}]},"finishReason":"STOP"}]}
The Gemini protocol has no [DONE] sentinel — the stream closing is the end signal, so don't wait for [DONE].
Parsing by hand (curl, or a client with no SDK)#
curl https://qd.zsopc.com/v1/chat/completions \
-H "Authorization: Bearer $PLATFORM_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-6-luna",
"stream": true,
"messages": [{"role":"user","content":"Hello"}]
}' \
--no-buffer | while read line; do
echo "$line"
done
Parsing with JavaScript fetch:
const resp = await fetch("https://qd.zsopc.com/v1/chat/completions", {
method: "POST",
headers: {
"Authorization": `Bearer ${process.env.PLATFORM_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "gpt-6-luna",
stream: true,
messages: [{ role: "user", content: "Hello" }],
}),
});
const reader = resp.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
while (true) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split("\n");
buffer = lines.pop() ?? "";
for (const line of lines) {
if (!line.startsWith("data: ")) continue;
const data = line.slice(6);
if (data === "[DONE]") return;
const chunk = JSON.parse(data);
process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}
}
Getting token usage#
OpenAI Chat streaming does not return a trailing usage chunk by default. To get real token counts, add stream_options:
stream = client.chat.completions.create(
model="gpt-6-luna",
messages=[...],
stream=True,
stream_options={"include_usage": True}, # the important bit
)
last_chunk = None
for chunk in stream:
if chunk.choices:
print(chunk.choices[0].delta.content or "", end="")
last_chunk = chunk
# the final chunk carries usage
print(f"\nTokens: {last_chunk.usage}")
Internally the gateway auto-injects stream_options.include_usage=true for most upstream channels (except GLM, which sends a usage chunk anyway). So you'll get token counts without setting it — but setting it explicitly is more robust.
Interrupting and cancelling#
OpenAI SDK (Python)#
stream = client.chat.completions.create(..., stream=True)
try:
for chunk in stream:
if some_user_canceled():
stream.close() # close explicitly; the gateway sees a client disconnect
break
print(chunk.choices[0].delta.content or "", end="")
except KeyboardInterrupt:
stream.close()
fetch (JavaScript)#
const controller = new AbortController();
const resp = await fetch("https://qd.zsopc.com/v1/chat/completions", {
signal: controller.signal,
// ...
});
// then, to interrupt:
controller.abort();
When the client disconnects, the gateway's upstream connection closes immediately too (on both HTTP and WebSocket channels) — the mechanism is a dropped connection rather than an explicit cancel instruction, and the effect is that the upstream stops generating. Tokens already generated are still billed; if the disconnect means the trailing usage chunk never arrived, the gateway books output tokens estimated from the characters already streamed.
Error handling#
Before the stream opens: an ordinary HTTP error, not SSE#
Auth failure (401), exhausted balance (402), a nonexistent model (400 model_not_found) and no available channel (503) all happen before the stream opens and come back as an ordinary HTTP JSON error body — there's no SSE yet, so handle them like non-streaming errors. Full list: Error handling.
After the stream opens (HTTP 200): it depends on the path#
Tool requests (web_search / image_generation, on the WS V2 channel): a mid-stream error is synthesised into an error chunk followed by [DONE]:
data: {"error":{"message":"<the specific reason>","type":"api_error","code":"upstream_error"}}
data: [DONE]
type and code are fixed values (api_error / upstream_error), so don't branch on code to classify the error — the actual reason is only in the message text. When parsing, check for an error field before handling delta.content, and treat error as the end-of-turn signal.
Plain text chat and every /v1/messages, /v1/responses and /v1beta path: when the upstream fails mid-stream the gateway synthesises nothing and the stream simply ends early — there's no guarantee of [DONE], finish_reason, message_stop or any error frame. A robust client must treat "stream ended without a terminator" as an interruption, not a normal completion. (Error events native to the upstream protocol, such as Anthropic's event: error, are forwarded verbatim. The gateway also generates error events itself in the keepalive case described under Long requests and timeouts.)
On either path, whatever content streamed before the error or interruption is partially valid and can be kept on screen.
Long requests and timeouts#
There is no 180-second total ceiling. The limits on one upstream call are:
| Case | Limit |
|---|---|
| Non-streaming request | 600 seconds |
Streaming request (stream: true), whole stream | about 61 minutes (3660 seconds) |
| Streaming request, waiting for the upstream to start responding | 120 seconds (not applied on the Gemini streaming endpoint) |
When the upstream sends nothing at all for 300 seconds in the middle of a stream, most channels treat it as gone and end the stream. Retriable failures switch channels automatically, but only within the first 60 seconds of the request, and one request tries at most 3 channels.
So that proxies along the way don't drop a connection that has carried no data for a while, the gateway answers long requests with HTTP 200 first and then keeps the connection alive:
- Streaming (
/v1/chat/completions,/v1/messages,/v1/responses): if the upstream hasn't started responding 20 seconds after the request arrived, the gateway sends 200 withtext/event-stream, then an SSE comment line: keepaliveevery 15 seconds. Standard SSE parsers ignore comment lines. - Non-streaming (those three endpoints and
/v1beta/...:generateContent): if the call hasn't finished after 20 seconds, the gateway sends 200, writes one space every 15 seconds, and then the full JSON body (JSON allows leading whitespace). These responses carry the headerx-gpushare-nonstream-keepalive: 1.
The cost: a failure after the 20-second mark can't return a 5xx status. A streaming request gets an in-stream error event instead (Chat: a data: line with error followed by [DONE]; Responses: response.failed; Anthropic: event: error). A non-streaming request gets HTTP 200 with the error in the body's error field. Decide success from the body, not from the status code alone. The Gemini streaming endpoint (:streamGenerateContent) has no keepalive before the stream opens.
What to do:
- Use streaming for long outputs; tokens already produced are billed as usual
- If your client enforces a deadline on the whole request, give non-streaming requests at least 660 seconds (the 600-second limit plus room for a channel switch). Streaming time grows with the length of the output, so time out on "no data received for a while" rather than on a fixed total
Per-endpoint timeouts: API reference.
Reverse proxies and Nginx#
The gateway sets these response headers:
Content-Type: text/event-stream
Cache-Control: no-cache, no-store, no-transform
X-Accel-Buffering: no
If you proxy this gateway behind your own reverse proxy, make sure buffering is off (proxy_buffering off; in Nginx) — otherwise the whole streamed response is buffered until it completes and delivered in one go, which defeats the point.
Debugging#
Watching the raw SSE with curl is the most direct approach:
curl https://qd.zsopc.com/v1/chat/completions \
-H "Authorization: Bearer $PLATFORM_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"gpt-6-luna","stream":true,"messages":[{"role":"user","content":"hi"}]}' \
--no-buffer 2>&1 | head -20
If you see the response arrive all at once instead of chunk by chunk, something in between is buffering.