# Streaming

> Parsing SSE, the usage chunk, cancellation, streaming errors, timeouts and keepalives on long requests, and when stream=true is required

Source: https://qd.zsopc.com/en/docs/guides/streaming

The platform supports SSE (Server-Sent Events) streaming. **The first token arrives sooner**, so your UI can render as generation happens.

## Basic usage

Client setup (base_url and API key) is covered in the [Quickstart](../quickstart.md); each snippet below repeats only the minimum.

### OpenAI SDK

```python
from openai import OpenAI

client = OpenAI(
    base_url="https://qd.zsopc.com/v1",
    api_key="sk-gpushare-xxx",
)

stream = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Stream a haiku"}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)
```

### Anthropic SDK

```python
from anthropic import Anthropic

client = Anthropic(
    base_url="https://qd.zsopc.com",  # ⚠️ no /v1
    api_key="sk-gpushare-xxx",
)

with client.messages.stream(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Stream a haiku"}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
```

### Gemini SDK

```python
from google import genai

client = genai.Client(
    api_key="sk-gpushare-xxx",
    http_options={"base_url": "https://qd.zsopc.com", "api_version": "v1beta"},
)

stream = client.models.generate_content_stream(
    model="gemini-2.5-pro",
    contents="Stream a haiku",
)
for chunk in stream:
    print(chunk.text, end="", flush=True)
```

## When stream=true is required

| Tool | Streaming required | Why |
|---|---|---|
| `image_generation` | ✅ | the upstream WebSocket channel has no synchronous response — it only streams |
| `web_search` | ✅ | same, and the search progress comes back live as stream chunks |
| `function` tools | either | they use the HTTP channel, streaming or not |
| Plain text chat | either | streaming or not |

Otherwise the gateway returns **400 `invalid_request_error`** with the message `` Tools `web_search` and `image_generation` require `stream: true` ``.

## SSE wire format

The gateway forwards the upstream SSE as-is: **each `data:` line is one chunk**.

### OpenAI Chat stream chunks

```
data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"}}]}

data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" world"}}]}

data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":10,"completion_tokens":2,"total_tokens":12}}

data: [DONE]
```

Two things hand-written parsers get wrong most often:

- The `finish_reason` chunk **carries no usage** — usage arrives as a **separate trailing chunk** right after it, which is the standard `stream_options.include_usage` convention
- The usage chunk's `choices` is an **empty array**, so `chunk.choices[0].delta` goes out of bounds — check `chunk.choices` is non-empty before reading `delta`

### Anthropic stream events

```
event: message_start
data: {"type":"message_start","message":{...}}

event: content_block_delta
data: {"type":"content_block_delta","delta":{"type":"text_delta","text":"Hello"}}

event: message_stop
data: {"type":"message_stop"}
```

### Gemini streaming (SSE)

```
data: {"candidates":[{"content":{"parts":[{"text":"Hello"}]}}]}

data: {"candidates":[{"content":{"parts":[{"text":" world"}]},"finishReason":"STOP"}]}
```

The Gemini protocol has **no `[DONE]` sentinel** — the stream closing is the end signal, so don't wait for `[DONE]`.

## Parsing by hand (curl, or a client with no SDK)

```bash
curl https://qd.zsopc.com/v1/chat/completions \
  -H "Authorization: Bearer $PLATFORM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-6-luna",
    "stream": true,
    "messages": [{"role":"user","content":"Hello"}]
  }' \
  --no-buffer | while read line; do
    echo "$line"
  done
```

Parsing with JavaScript fetch:

```javascript
const resp = await fetch("https://qd.zsopc.com/v1/chat/completions", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.PLATFORM_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "gpt-6-luna",
    stream: true,
    messages: [{ role: "user", content: "Hello" }],
  }),
});

const reader = resp.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });

  const lines = buffer.split("\n");
  buffer = lines.pop() ?? "";

  for (const line of lines) {
    if (!line.startsWith("data: ")) continue;
    const data = line.slice(6);
    if (data === "[DONE]") return;
    const chunk = JSON.parse(data);
    process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  }
}
```

## Getting token usage

OpenAI Chat streaming does **not** return a trailing usage chunk by default. To get real token counts, add `stream_options`:

```python
stream = client.chat.completions.create(
    model="gpt-6-luna",
    messages=[...],
    stream=True,
    stream_options={"include_usage": True},  # the important bit
)

last_chunk = None
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="")
    last_chunk = chunk

# the final chunk carries usage
print(f"\nTokens: {last_chunk.usage}")
```

Internally the gateway **auto-injects** `stream_options.include_usage=true` for most upstream channels (except GLM, which sends a usage chunk anyway). So you'll get token counts without setting it — but setting it explicitly is more robust.

## Interrupting and cancelling

### OpenAI SDK (Python)

```python
stream = client.chat.completions.create(..., stream=True)
try:
    for chunk in stream:
        if some_user_canceled():
            stream.close()  # close explicitly; the gateway sees a client disconnect
            break
        print(chunk.choices[0].delta.content or "", end="")
except KeyboardInterrupt:
    stream.close()
```

### fetch (JavaScript)

```javascript
const controller = new AbortController();

const resp = await fetch("https://qd.zsopc.com/v1/chat/completions", {
  signal: controller.signal,
  // ...
});

// then, to interrupt:
controller.abort();
```

When the client disconnects, the gateway's upstream connection **closes immediately too** (on both HTTP and WebSocket channels) — the mechanism is a dropped connection rather than an explicit cancel instruction, and the effect is that the upstream stops generating. **Tokens already generated are still billed**; if the disconnect means the trailing usage chunk never arrived, the gateway books output tokens estimated from the characters already streamed.

## Error handling

### Before the stream opens: an ordinary HTTP error, not SSE

Auth failure (401), exhausted balance (402), a nonexistent model (400 `model_not_found`) and no available channel (503) all happen **before the stream opens** and come back as an ordinary HTTP JSON error body — there's no SSE yet, so handle them like non-streaming errors. Full list: [Error handling](../reference/errors.md).

### After the stream opens (HTTP 200): it depends on the path

**Tool requests (`web_search` / `image_generation`, on the WS V2 channel)**: a mid-stream error is synthesised into an error chunk followed by `[DONE]`:

```
data: {"error":{"message":"<the specific reason>","type":"api_error","code":"upstream_error"}}

data: [DONE]
```

`type` and `code` are **fixed values** (`api_error` / `upstream_error`), so don't branch on `code` to classify the error — the actual reason is only in the `message` text. When parsing, **check for an `error` field before handling `delta.content`**, and treat `error` as the end-of-turn signal.

**Plain text chat and every `/v1/messages`, `/v1/responses` and `/v1beta` path**: when the upstream fails mid-stream the gateway **synthesises nothing** and the stream simply **ends early** — there's no guarantee of `[DONE]`, `finish_reason`, `message_stop` or any error frame. A robust client must treat "stream ended without a terminator" as an interruption, not a normal completion. (Error events native to the upstream protocol, such as Anthropic's `event: error`, are forwarded verbatim. The gateway also generates error events itself in the keepalive case described under [Long requests and timeouts](#long-requests-and-timeouts).)

On either path, whatever content streamed before the error or interruption is partially valid and can be kept on screen.

### Long requests and timeouts

There is **no** 180-second total ceiling. The limits on one upstream call are:

| Case | Limit |
|---|---|
| Non-streaming request | 600 seconds |
| Streaming request (`stream: true`), whole stream | about 61 minutes (3660 seconds) |
| Streaming request, waiting for the upstream to start responding | 120 seconds (not applied on the Gemini streaming endpoint) |

When the upstream sends nothing at all for 300 seconds in the middle of a stream, most channels treat it as gone and end the stream. Retriable failures switch channels automatically, but only within the first 60 seconds of the request, and one request tries at most 3 channels.

So that proxies along the way don't drop a connection that has carried no data for a while, the gateway answers long requests with HTTP 200 first and then keeps the connection alive:

- **Streaming** (`/v1/chat/completions`, `/v1/messages`, `/v1/responses`): if the upstream hasn't started responding 20 seconds after the request arrived, the gateway sends 200 with `text/event-stream`, then an SSE comment line `: keepalive` every 15 seconds. Standard SSE parsers ignore comment lines.
- **Non-streaming** (those three endpoints and `/v1beta/...:generateContent`): if the call hasn't finished after 20 seconds, the gateway sends 200, writes one space every 15 seconds, and then the full JSON body (JSON allows leading whitespace). These responses carry the header `x-gpushare-nonstream-keepalive: 1`.

The cost: **a failure after the 20-second mark can't return a 5xx status**. A streaming request gets an in-stream error event instead (Chat: a `data:` line with `error` followed by `[DONE]`; Responses: `response.failed`; Anthropic: `event: error`). A non-streaming request gets HTTP 200 with the error in the body's `error` field. Decide success from the body, not from the status code alone. The Gemini streaming endpoint (`:streamGenerateContent`) has no keepalive before the stream opens.

What to do:

- Use streaming for long outputs; tokens already produced are billed as usual
- If your client enforces a deadline on the whole request, give non-streaming requests at least 660 seconds (the 600-second limit plus room for a channel switch). Streaming time grows with the length of the output, so time out on "no data received for a while" rather than on a fixed total

Per-endpoint timeouts: [API reference](../reference/api-reference.md).

## Reverse proxies and Nginx

The gateway sets these response headers:
```http
Content-Type: text/event-stream
Cache-Control: no-cache, no-store, no-transform
X-Accel-Buffering: no
```

If you proxy this gateway behind your own reverse proxy, make sure **buffering is off** (`proxy_buffering off;` in Nginx) — otherwise the whole streamed response is buffered until it completes and delivered in one go, which defeats the point.

## Debugging

Watching the raw SSE with curl is the most direct approach:

```bash
curl https://qd.zsopc.com/v1/chat/completions \
  -H "Authorization: Bearer $PLATFORM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-6-luna","stream":true,"messages":[{"role":"user","content":"hi"}]}' \
  --no-buffer 2>&1 | head -20
```

If you see the response arrive **all at once** instead of chunk by chunk, something in between is buffering.
