# Streaming

Set `"stream": true` and both surfaces stream Server-Sent Events (SSE): the
OpenAI-compatible surface at `POST https://dev-api.routerplus.com/v1/chat/completions` and the
Anthropic-compatible surface at `POST https://dev-api.routerplus.com/v1/messages`.

The images route (`POST https://dev-api.routerplus.com/v1/images/generations`) does not stream:
`stream: true` and `partial_images` are typed 400s there.

**Same-dialect routes relay.** When the provider serving your model speaks your
surface's dialect (a Claude model on `/v1/messages`, a GPT model on
`/v1/chat/completions`), the provider's stream is relayed record-for-record with
payloads untouched — provider-specific features (server tool use, web search
results, citations, annotations, `pause_turn`, the matched `stop_sequence`)
reach you exactly as if you called the provider directly. The gateway only
observes (for billing and failover), injects `usage.cost` into the provider's
own usage record, and guarantees a well-formed terminator. On the OpenAI
surface each chunk also keeps OpenAI's fields only and names the model you
asked for: a field the provider adds of its own (its cost, its accounting, its
host's name) is dropped, and the delta passes as the provider sent it (see
[what an answer keeps](/docs/compat#what-an-answer-from-an-openai-dialect-provider-keeps)).
One exception: an explicit `stream_options: {"include_usage": false}` turns the
relay off, because the gateway must then remove the usage chunk it asked the
provider for.

**Cross-dialect routes re-encode.** When the provider speaks the other dialect,
the gateway translates its stream into your surface's exact wire framing, and
the shapes below hold.

Before the first byte there is a 20-second headers deadline — a provider that
can't answer in time is failed over invisibly (see
[Routing & failover](/docs/routing)). After that a stream is not cut for being
slow, but every request has a deadline: 450 seconds from dispatch, or the
`timeout_ms` of a [routing policy](/docs/routing-policies) (1–600 s). A stream
still open at the deadline ends with the terminal error event described below.

```bash
curl -N https://dev-api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"claude-sonnet-5","stream":true,"max_tokens":100,
       "messages":[{"role":"user","content":"Say hello"}]}'
```

Every stream response carries `x-request-id`, `x-tm-provider` (the deployment
that served you), `x-tm-attempts`, and `x-tm-upstream-status` headers, plus
`x-tm-dropped-params`, `x-tm-upstream-model` and `x-tm-served-by` when they
apply (see [Wire compatibility](/docs/compat#response-headers)).

## Frame order — OpenAI surface

Each frame is a complete `data: {json}\n\n` line in the documented
`chat.completion.chunk` shape. The order is fixed: role-priming delta, content
deltas, finish chunk, usage chunk, `data: [DONE]`.

```text
data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"content":" there."},"finish_reason":null}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{},"finish_reason":"stop","native_finish_reason":"end_turn"}]}

data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":9,"total_tokens":21,"prompt_tokens_details":{"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"reasoning_tokens":0},"cost":0.000114}}

data: [DONE]
```

Notes on this surface:

- The **role-priming delta** (`{"role":"assistant","content":""}`) is emitted as
  soon as the upstream proves alive — it is a liveness signal, not content.
- Tool calls stream as `delta.tool_calls` entries with stable ascending
  `index` values; the arguments arrive as string fragments to concatenate. A
  tool called with no arguments gets one `"{}"` fragment.
- Reasoning models stream their thinking as `delta.reasoning_content`.
- Refusal text streams as `delta.refusal`.
- On a translated stream the finish chunk also carries
  `native_finish_reason`, the provider's own stop reason (here `end_turn`).
- The usage chunk is always sent, zero-filled if the provider reported nothing,
  unless you sent `stream_options.include_usage: false`.
- `[DONE]` is only ever sent after a successful finish — **never after an
  error** (see below).

## Frame order — Anthropic surface

Named SSE events (`event: <name>\ndata: {json}\n\n`) exactly as the Anthropic
Messages API frames them: `message_start`, `content_block_start` /
`content_block_delta` / `content_block_stop`, `message_delta`, `message_stop`.

```text
event: message_start
data: {"type":"message_start","message":{"id":"msg_2f6d8e3b","type":"message","role":"assistant","model":"gpt-4o-mini","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":0,"cache_read_input_tokens":0,"cache_creation_input_tokens":0,"output_tokens":0}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello there."}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"input_tokens":12,"cache_read_input_tokens":0,"cache_creation_input_tokens":0,"output_tokens":9,"cost":0.000007}}

event: message_stop
data: {"type":"message_stop"}
```

Notes on this surface:

- On a translated stream the token counts in `message_start` are zeros; the
  authoritative usage (and `cost`) arrives in the final `message_delta`. Read
  usage there, not from `message_start`. On a relayed stream `message_start`
  carries the provider's own counts.
- Tool calls stream as `tool_use` content blocks with `input_json_delta`
  fragments; thinking streams as `thinking` blocks with `thinking_delta` (and
  `signature_delta` where the provider signs it).
- `input_tokens` excludes cache reads and writes, per the
  [billing contract](/docs/compat) — cache traffic is reported separately in
  `cache_read_input_tokens` and `cache_creation_input_tokens`.

## Usage is always in the stream

You never opt in to stream usage. The gateway injects
`stream_options.include_usage` into every OpenAI-dialect upstream dispatch
itself (without it, OpenAI streams carry no usage at all). An explicit
`stream_options: {"include_usage": false}` is honored: you are billed the same,
but no usage chunk is emitted to you. Every other stream ends
with provider-reported token counts, and every **billed** stream carries
`usage.cost` (USD) — computed with the exact integer micro-USD math the ledger
settles with, covering the full request debit including any attempts that
failed over before your answer started.

> [!TIP]
> `cost` on the wire is the number you are charged. You can recompute it any
> time from the token counts and the public prices at
> [https://dev.routerplus.com/api/models.json](https://dev.routerplus.com/api/models.json) — settlement
> rounds down, in your favor.

## Keep-alive frames

Slow reasoning models can go quiet for a long time mid-answer, and idle TCP
connections get killed by proxies, load balancers, and some HTTP clients. So
once your stream has started, any silence of 15 seconds or more produces a
keep-alive frame:

- **OpenAI surface:** an SSE comment — `: processing` — which the SSE spec
  requires clients to ignore. SDKs never see it.
- **Anthropic surface:** a native ping event —
  `event: ping` / `data: {"type":"ping"}` — the same frame Anthropic's own API
  sends, which Anthropic SDKs already skip.

Keep-alives are measured from the last real write, so the silence you can
observe is bounded at ~15s, and nothing — keep-alives included — is ever sent
after a terminal event.

## Cancelling a stream

Close the connection (abort the HTTP request) and the gateway immediately
aborts the upstream request, so the provider stops generating.

The billing semantics are deliberate and worth knowing:

- **You are billed for the input and what was streamed to you** up to the
  abort — the attempt is recorded with outcome `cancelled`, visible at
  `GET /v1/generation?id=<x-request-id>`. If the provider reported no final
  usage, you pay for the input (the provider's count when it sent one, else the
  gateway's estimate) and the output you received, never the full hold. A
  cancel before the first token costs the input only.
- **A cancellation never counts against the provider's uptime.** You hung up;
  the provider did nothing wrong. Cancelled attempts are excluded from the
  public uptime numbers, and from the health circuits too, except one: when
  the provider had sent no response headers for 60 seconds before you hung up,
  it was leaving the request hanging, and that counts as its failure (see
  [Routing & failover](/docs/routing#health-circuits-and-cooldowns)).

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://dev-api.routerplus.com/v1",
  apiKey: process.env.TM_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "claude-sonnet-5",
  max_tokens: 4096,
  messages: [{ role: "user", content: "Explain SSE, thoroughly." }],
  stream: true,
});

let chars = 0;
for await (const chunk of stream) {
  chars += chunk.choices[0]?.delta?.content?.length ?? 0;
  if (chars > 2000) {
    stream.controller.abort(); // upstream generation stops; billed to here
    break;
  }
}
```

## When a provider dies mid-stream

If the serving provider fails **after** your answer started, the stream does
not silently end and the request is never restarted on another provider (the
commit boundary — see [Routing & failover](/docs/routing)). Instead you get one
terminal error event inside the already-open 200 stream:

**OpenAI surface** — any known usage (and its `cost`) is emitted first, so the
billed partial attempt stays recomputable from the wire, then a final chunk
with `finish_reason: "error"` and a top-level `error`:

```text
data: {"id":"chatcmpl-2f6d8e3b","object":"chat.completion.chunk","created":1757000000,"model":"claude-sonnet-5","choices":[{"index":0,"delta":{"content":""},"finish_reason":"error"}],"error":{"code":"upstream_error","message":"upstream failure","type":"gateway_error","metadata":{"error_type":"upstream_error"}}}
```

**Anthropic surface** — a native `error` event carrying both the
Anthropic-native type string and the stable canonical `error_type`:

```text
event: error
data: {"type":"error","error":{"type":"api_error","message":"upstream failure","error_type":"upstream_error"}}
```

Nothing follows a terminal event — no `[DONE]`, no further frames. The same
event ends a stream that reaches the request deadline, and a stream whose
provider closed the connection without a proper terminator.

> [!NOTE]
> The Anthropic wire has no legal usage slot outside `message_delta`, so an
> Anthropic-surface stream that dies mid-answer cannot carry its partial usage
> in-band. `GET /v1/generation?id=<x-request-id>` returns the settled numbers
> for exactly this case. This is a documented limitation of the wire format,
> not of the ledger.

The full taxonomy of `error_type` values and what to do about each lives in
[Errors & remediation](/docs/errors).

## SDK streaming examples

Both official SDKs work unmodified — swap the base URL and key.

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://dev-api.routerplus.com/v1", api_key=os.environ["TM_API_KEY"])

stream = client.chat.completions.create(
    model="claude-sonnet-5",
    max_tokens=200,
    messages=[{"role": "user", "content": "Write a haiku about failover."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:  # the final usage chunk has an empty choices list
        print(f"\ncost: ${chunk.usage.cost}")
```

```python
import os
import anthropic

client = anthropic.Anthropic(base_url="https://dev-api.routerplus.com", api_key=os.environ["TM_API_KEY"])

with client.messages.stream(
    model="gpt-4o-mini",  # yes — a GPT model over the Anthropic wire
    max_tokens=200,
    messages=[{"role": "user", "content": "Write a haiku about failover."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
    final = stream.get_final_message()
    print(f"\nstop_reason: {final.stop_reason}")
```