Streaming an LLM Response: SSE, Partial Render, Cancel

Streaming an LLM response token by token to the screen raises an immediate question: how do you actually implement LLM streaming with SSE, and does it really change the user experience? The short answer is yes — but opening the SSE connection is only the start. Rendering partial Markdown, cleanly cancelling a request with AbortController, and making sure a reverse proxy like nginx doesn't silently buffer the stream make up the bulk of the real work. In this article we build SSE end to end starting from the official streaming behavior of the Claude Messages API and OpenAI, and walk step by step through partial rendering and cancellation mechanics.

💡 Pro Tip: While testing the stream, watch the response in the "EventStream" view under the Network tab in your browser's DevTools — this is where you actually see whether chunks are arriving piece by piece or all at once, and you'll catch a proxy buffering problem here far faster than by scrolling through console.log lines.

Table of Contents

• Perceived latency: TTFT vs total duration
• Comparing SSE, WebSocket, and chunked responses
• Server side: proxy, buffer, and timeout traps
• Rendering partial Markdown/code blocks
• Cancellation and stopping cost with AbortController
• Errors, reconnection, and partial response recovery
• Streaming on mobile (Flutter/Dart)
• FAQ
• How do you stream an LLM response?
• Should I use SSE or WebSocket?
• Does token cost stop when the user cancels?
• How do you safely render Markdown/JSON while streaming?
• Why does SSE seem to not work behind nginx?
• Update (September 2026)
• Conclusion
• Sources

Perceived latency: TTFT vs total duration

The latency a user feels while waiting for an LLM response depends far less on the model's total generation time than on the moment the first token lands on screen (time-to-first-token, TTFT). With streaming off, the client sees nothing until the model finishes the entire response; with streaming on, the first chunk arrives much earlier and the interface signals "the system is working" right away.

OpenAI's own cookbook example shows this concretely: "with the streaming request, we received the first token after 0.1 seconds, and subsequent tokens every ~0.01-0.02 seconds." These numbers come from OpenAI's own sample run, not a general performance guarantee, but they show why TTFT needs to be measured separately from total duration.

The API surface reflects this too: in OpenAI's streaming responses, each chunk carries a delta field instead of a full message field — "delta can hold things like: a role token (e.g., {\"role\": \"assistant\"}), a content token (e.g., {\"content\": \"\\n\\n\"}), nothing (e.g., {}), when the stream is over". To see the usage field you need stream_options={"include_usage": true}; once set, usage comes back null on every chunk except the last, where the whole request's token count finally appears. Practical consequence: your own token/cost counter can't trust intermediate chunks — wait for the final chunk or the provider's cumulative counter event (in Claude, message_delta.usage, see the next section).

Comparing SSE, WebSocket, and chunked responses

You have three options for setting up a one-way data stream from server to client: Server-Sent Events (SSE), WebSocket, and fetch's raw ReadableStream body. Nearly every LLM provider picks SSE, because the need is usually one-directional: the server talks, the client listens.

As MDN defines it, with SSE "it's possible for a server to send new data to a web page at any time, by pushing messages to the web page." The EventSource interface abstracts this in the browser and gives you two behaviors for free: "By default, if the connection between the client and server closes, the connection is restarted" (automatic reconnection), and the server can suggest this reconnection delay in milliseconds via the retry: field.

WebSocket, by contrast, "makes it possible to open a two-way interactive communication session between the user's browser and a server," per MDN. Since client-to-server traffic in an LLM chat is usually a single request (the prompt), the bidirectionality WebSocket brings is often unnecessary complexity.

The third option is reading directly over fetch without ever touching SSE's protocol layer: "the body read-only property of the Response interface is a ReadableStream of the body contents" — the Response.body returned by fetch() is a stream readable chunk by chunk. Some SDKs stream their own JSON-lines format over the raw chunked body without the SSE envelope (event:/data: lines); the logic is the same, only the envelope differs.

SSE's known limitation is also documented on MDN: "SSE suffers from a limitation to the maximum number of open connections... the limit is per browser and is set to a very low number (6)" — without HTTP/2, this ceiling can bite in heavily multi-tab scenarios; the fix is to terminate the server over HTTP/2 (or HTTP/3).

Server side: proxy, buffer, and timeout traps

The most common reason SSE "looks like it's not working" isn't the client code — it's the reverse proxy sitting in between. The nginx documentation states the default plainly: "nginx receives a response from the proxied server as soon as possible, saving it into the buffers set by the proxy_buffer_size and proxy_buffers directives" — nginx by default collects the upstream response into buffers and delivers it once full, not chunk by chunk. Result: your code is correct, but the response in the browser still arrives "all at once."

The fix is one of two paths:

1location /api/stream {
2    proxy_pass http://upstream_llm;
3    proxy_buffering off;
4    proxy_read_timeout 3600s;
5}

The nginx documentation describes the proxy_buffering off behavior as follows: "the response is passed to a client synchronously, immediately as it is received" — the response passes to the client synchronously, as soon as it's received. If you want this control for a single response rather than the whole location, you can also set it from the upstream application's header: "Buffering can also be enabled or disabled by passing 'yes' or 'no' in the 'X-Accel-Buffering' response header field" — meaning the backend itself can turn off buffering by returning an X-Accel-Buffering: no header, without touching the nginx configuration.

The second trap is around proxy_read_timeout: "The timeout is set only between two successive read operations, not for the transmission of the whole response" — this duration limits the gap between two consecutive reads, not the total response time. If the model goes quiet for a while while preparing a long tool call (say, while building a large tool_use block), the connection can close because of this silent gap even before the total duration runs out; that's why you need to raise proxy_read_timeout to a reasonable value on streaming endpoints.

Rendering partial Markdown/code blocks

Every text_delta you see on the client side isn't a meaningful Markdown/code fragment on its own — it can be cut off mid code-fence or mid table row. Anthropic's docs describe a similar problem for tool input (tool_use): "the deltas are partial JSON strings, whereas the final tool_use.input is always an object." The same principle generalizes to text rendering: accumulate the raw deltas in a buffer, re-parse the entire buffer on every new chunk, and write only the final result to the DOM.

1let buffer = "";
2 
3function onTextDelta(delta: string) {
4  buffer += delta;
5  // Re-parse the ENTIRE accumulated text on every delta,
6  // never write raw broken Markdown directly to the DOM.
7  const html = renderMarkdownSafely(buffer);
8  articleEl.innerHTML = html;
9}
10 
11function onContentBlockStop() {
12  // Draw the final, complete version once more when the block ends.
13  articleEl.innerHTML = renderMarkdownSafely(buffer);
14}

The same "accumulate, merge at the boundary" logic applies to streams returning structured output (JSON Schema); we covered that topic in depth in Getting structured output from an LLM: JSON Schema — the real difference here is that partial JSON is never shown on screen and is only collected in the background, whereas with partial Markdown the user sees it, so every delta requires a re-render. A practical rule for detecting code fences: if the number of code fence markers (triple backticks) in the accumulated text is odd, rendering the last fence "as if closed" and leaving the closing marker open until it arrives makes an unfinished code block look more readable to the user.

Cancellation and stopping cost with AbortController

When the user says "stop," the tool you need is already sitting in the browser: AbortController. MDN's definition of AbortController.abort() says this API "Aborts an asynchronous operation before it has completed. This is able to abort fetch requests, the consumption of any response bodies, or streams" — meaning you can stop both an ongoing fetch request and body consumption (and therefore stream reading) with a single call.

1let controller: AbortController | null = null;
2const decoder = new TextDecoder();
3 
4async function streamAnswer(prompt: string) {
5  controller = new AbortController();
6  const res = await fetch("/api/stream", {
7    method: "POST",
8    body: JSON.stringify({ prompt }),
9    signal: controller.signal,
10  });
11 
12  const reader = res.body!.getReader();
13  while (true) {
14    const { done, value } = await reader.read();
15    if (done) break;
16    onTextDelta(decoder.decode(value, { stream: true }));
17  }
18}
19 
20// When the user clicks the "stop" button:
21cancelButton.addEventListener("click", () => controller?.abort());

Be aware of a boundary here: controller.abort() definitively stops the connection and read loop on the client side, but whether the provider's server stops generation at that moment, or bills for the tokens already produced, isn't clearly documented by Anthropic or OpenAI. Treat this as provider-specific; don't assume "cancelled means billing stopped instantly" — closing the connection is guaranteed, what the server does about it is not.

Errors, reconnection, and partial response recovery

SSE's own protocol already offers a contract for errors and partial recovery. If an error occurs during a Claude Messages API stream, it's sent as a separate event:

1event: error
2data: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}}

Think of this as the in-stream equivalent of an HTTP 529 error — the connection stays open, and the error passes through as an event. With EventSource on the client side, it reconnects automatically when the connection drops, and the server can suggest the wait time via retry:: "the browser will wait for the specified time before attempting to reconnect." With your own fetch+ReadableStream implementation, you must build this reconnection by hand.

Be careful about resuming from where you left off: neither Anthropic's nor OpenAI's documentation officially guarantees "the connection dropped, resume from the last token"; in practice clients usually resend the request from scratch. Design your reconnection code accordingly — "SSE reconnects automatically" and "the request resumes from where it left off" are two different claims, and only the first one is officially guaranteed.

Streaming on mobile (Flutter/Dart)

Consuming a stream on the Flutter/Dart side is a natural counterpart to the ReadableStream logic on the web. Dart's official documentation says this about HttpClientResponse: "The body of an HttpClientResponse object is a Stream of data from the server," and among the types the class implements is Stream<List<int>> directly. In other words, the incoming HTTP response can be processed chunk by chunk with standard Stream APIs like transform/listen:

1final request = await httpClient.getUrl(uri);
2final response = await request.close();
3 
4final buffer = StringBuffer();
5await for (final chunk in response.transform(utf8.decoder)) {
6  buffer.write(chunk);
7  onPartialText(buffer.toString());
8}

FAQ

How do you stream an LLM response?

Calling the model on the server side with the stream: true parameter (Claude Messages API) or stream=True (OpenAI) makes it return the response chunk by chunk as an SSE event sequence with the text/event-stream content type; on the client side these events are then consumed either with EventSource or with the ReadableStream read from fetch's Response.body.

Should I use SSE or WebSocket?

If you're only sending a one-way text stream from server to client (the standard scenario for an LLM chat), SSE is enough and simpler: EventSource gives you automatic reconnection and the retry field for free. If you also need to send continuous, low-latency data from client to server (e.g., a shared cursor position, live audio), WebSocket's two-way model is a better fit.

Does token cost stop when the user cancels?

On the client side, AbortController.abort() definitively stops the connection and the read loop. However, whether the provider bills for the tokens produced up to that point, or whether the model keeps generating on the server side, isn't clearly stated in the official Anthropic/OpenAI documentation; treat this as provider-specific behavior that needs to be verified.

How do you safely render Markdown/JSON while streaming?

Never write raw deltas directly to the DOM. Instead, accumulate all the chunks in a string buffer, run the entire accumulated text through a safe Markdown/JSON parser on every new chunk, and print only the converted result to the screen; do one final full render when the block completes (content_block_stop).

Why does SSE seem to not work behind nginx?

Most likely, nginx's default proxy_buffering on behavior is collecting the whole response before sending it. Adding proxy_buffering off to the location block, or having the backend return an X-Accel-Buffering: no header, makes the response pass to the client synchronously as soon as it's received.

Update (September 2026)

The body of this article describes the core SSE/EventSource mechanics, nginx buffering behavior, and OpenAI's stream=True contract as of 2026-06-09; these core mechanics haven't changed since then. However, two developments that directly touch on partial rendering and cancellation are noted here, since they can be verified against official sources.

On June 30, 2026, an update landed for Claude's agent session stream (GET /v1/sessions/{id}/events/stream) adding chunk-by-chunk preview: agent message text can now be streamed piece by piece in advance instead of as a single event — a sign that the accumulate-and-re-render pattern from the "partial rendering" section of this article extends to non-chat agent streams too. Source: platform.claude.com/docs/en/release-notes/api.

On September 1, 2026, the thinking.display: "updates" option (beta) was released for the thinking/reasoning stream: instead of streaming the full thinking text, the model can now stream only progress updates between tool calls. This is a new option the provider added to the question of "how much should I show the user during the stream." Source: platform.claude.com/docs/en/build-with-claude/streaming.

Conclusion

Streaming an LLM response with SSE isn't just a single stream: true parameter; the real work is in safely rendering partial content, correctly wiring up cancellation with AbortController, and making sure every reverse proxy in between (nginx, CDN) isn't buffering the stream. If you're producing tool_use output, you can find how to accumulate and parse partial JSON in Getting structured output from an LLM: JSON Schema; if you're wiring this stream into an agent loop, the planner pattern is in Agentic AI: tool use, planner loop, and production. If you're curious about the protocol underneath streaming itself, AI integration with MCP (Model Context Protocol) and Claude Code and MCP cover the underlying event-based communication model in a broader context. If you want to keep token cost under control while streaming is on, Cutting cost 10x with Prompt Caching is a complementary resource.

In practice, the part that breaks most often is almost never the client code — a reverse proxy's default buffering setting, or a timeout value, can make a streaming implementation that's been correctly written for months look "broken." Once you apply the checklist above, most of this class of bug disappears.

Sources

• Claude Messages API — Streaming Messages — the official definition of SSE event types, text_delta, partial JSON for tool_use, and message_delta.usage behavior.
• MDN — Server-sent events — the basic model of SSE.
• MDN — Using server-sent events — EventSource automatic reconnection, the retry field, and the browser connection limit.
• MDN — AbortController — cancelling fetch requests and stream consumption.
• MDN — AbortController.abort() — the exact definition of the abort() method.
• MDN — The WebSocket API — the definition of the two-way communication session.
• MDN — Response.body — reading the body as a ReadableStream.
• nginx — ngx_http_proxy_module — proxy_buffering, X-Accel-Buffering, proxy_read_timeout behavior.
• OpenAI Cookbook — How to stream completions — stream=True, the delta field, and the TTFT example.
• Dart — HttpClientResponse class — the response body as Stream<List<int>>.