AI Streaming Responses: Better UX, No Extra Cost

Your AI feature isn’t slow. It just feels slow.

A typical GPT-4o response takes 2–5 seconds end to end. If you’re displaying nothing until the full response is ready, that pause reads as broken. Users click refresh. They wonder if something failed. They rate the feature poorly in your next survey — not because the output was wrong, but because the wait was unbearable.

Streaming fixes this. Zero model changes, zero prompt rewrites, zero increase in API costs. You’re changing when the output arrives in the browser, not how much of it you get.

The Perception Gap Behind “Slow AI”

Time-to-first-token is what users actually feel

There are two numbers that matter in AI latency: time-to-first-token (TTFT) and total generation time. Most engineers optimize for total generation time. Users experience TTFT.

TTFT is how long it takes for the first characters to appear on screen. On GPT-4o, that’s typically 200–800ms. On Claude Sonnet, similar. Those numbers are fast enough that users don’t register them as delays — if you show them.

Without streaming, TTFT becomes total generation time. A 3-second response becomes a 3-second blank screen with a spinner. A 6-second response starts to feel broken. The actual quality of the output doesn’t matter if users have already decided the feature doesn’t work.

Why batching hurts even on fast requests

The instinct to wait for the full response usually comes from wanting to validate or format it before display. That’s a real concern — but it has better solutions than batching the entire response. Most validation can run on the completed buffer after streaming, not as a gate before display. And most formatting (markdown rendering, code highlighting) works incrementally.

The only cases where you genuinely need the full response before displaying anything are: structured JSON extraction that would show partial invalid JSON to the user, output that requires significant post-processing like translation or redaction, and features where the first token would be misleading without the rest (rare, and usually a prompt design problem).

How LLM Streaming Works

The server-sent events pattern

For web apps, server-sent events (SSE) are the standard approach. SSE is a long-lived HTTP connection where the server pushes data to the client as it becomes available. The client opens it once; the server keeps writing.

The pattern looks like this:

Client → POST /api/chat (sends message)
Server → opens SSE stream
Server → forwards each token from LLM API as it arrives
Client → renders tokens in real time
Server → sends [DONE] sentinel and closes stream

Every major LLM API supports streaming natively. Anthropic, OpenAI, and Google all emit tokens as server-sent events on their streaming endpoints. Your server acts as a relay — intercepting the stream from the upstream API and forwarding it to your frontend.

# Example: relaying Anthropic streaming to a frontend SSE response
async def stream_chat(message: str):
    async with anthropic.messages.stream(
        model="claude-sonnet-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": message}],
    ) as stream:
        async for text in stream.text_stream:
            yield f"data: {json.dumps({'token': text})}\n\n"
    yield "data: [DONE]\n\n"

The frontend reads this with the browser’s EventSource API or a fetch-based stream reader, appending each token to the display as it arrives.

WebSockets: when SSE isn’t enough

SSE is unidirectional — server to client only. That’s fine for most AI features. But if your feature involves back-and-forth within a single connected session — voice AI where the user can interrupt mid-response, collaborative document editing where multiple users contribute to the same AI session, or real-time AI that must receive sensor data while generating output — you need WebSockets.

WebSockets carry more overhead (connection upgrade, bidirectional channel management) and are harder to proxy through infrastructure like Cloudflare or AWS API Gateway. Reach for them only when you actually need bidirectionality. For the vast majority of chat, generation, and assist features, SSE is simpler and more reliable.

Streaming in Real Products

Chat and generation features: stream everything

For any feature where an AI writes prose — chat assistants, content generation, email drafters, code generators — streaming is the default. There’s no meaningful reason not to. The output is inherently sequential, partial tokens are legible, and TTFT is the primary user experience signal.

The one implementation detail worth getting right: buffer a short window of tokens before rendering. Rendering single characters creates a jarring flicker. Batching 4–8 tokens before flushing to the DOM gives smooth streaming without the per-character seizure.

Structured data extraction: think before you stream

If your feature extracts JSON from user input — filling a form, parsing a document, populating a database record — streaming the raw JSON output usually doesn’t make sense. Partial JSON is invalid JSON. Showing {"name": "Ac before the field completes isn’t useful to the user.

The right pattern here: stream to your server buffer, but only send the complete validated response to the client. Or stream partial results at the object level — show the first completed field as soon as its closing quote arrives — rather than at the character level. This gives you meaningful progressive display without exposing partial invalid states.

Long-form content: progressive disclosure

For features that generate long-form content (reports, documentation, structured plans), streaming enables progressive disclosure — showing the user completed sections as they’re generated rather than the full document all at once.

This works especially well with AI memory features where the model builds context incrementally. The user can start reading section one while section two is still generating. Total wait time is the same; perceived wait time is near-zero.

When to Skip Streaming

Background jobs and async processing never need streaming. If the AI is running a classification on uploaded documents overnight, or generating weekly summaries that land in a user’s inbox, there’s no one watching a screen. Stream to a queue, not to a socket.

Safety and compliance checks sometimes require full-output validation before display. If your product operates in healthcare, finance, or legal contexts where an incomplete AI statement could be acted on — and where AI guardrails require the full output before release — batch processing is correct. Display a loading state and reveal everything when it’s clean.

Making It Production-Ready

Streaming connections drop. Mobile users switch networks. Tabs get backgrounded. Your frontend needs graceful degradation: if the SSE connection closes before the [DONE] sentinel, show what was received plus a retry option. Don’t lose partial output silently.

On the infrastructure side, streaming requires responses that stay open for multiple seconds — which conflicts with serverless function timeout defaults. AWS Lambda’s default 29-second timeout is usually fine for generation tasks, but check that your API Gateway, load balancer, or CDN layer doesn’t have a shorter idle timeout that would terminate the stream early.

Streaming also pairs well with prompt caching — the two optimizations are independent and additive. Prompt caching reduces what you pay per request. Streaming reduces how long users wait for the first token. Together, they’re the foundation of a production AI feature that feels fast and doesn’t break the budget.

If you’re building AI-powered products and still displaying a spinner until the full response arrives, streaming is the highest-ROI change you haven’t made yet. It takes an afternoon to implement and permanently changes how users experience every AI feature you ship.

---

Yaniv Amrami is founder of quickdev. He has helped Israeli startups design and ship AI product features that feel fast and work reliably at scale.