Streaming: Why First-Token Time Beats Total Time

The latency post measured what the clock says. This post is about what the user feels — and the two disagree in a specific, exploitable way. A user staring at a blank screen experiences every second as an eternity; a user watching words appear experiences the same total time as progress. The clock is identical. The experience is not.

Streaming is the difference: instead of waiting for the complete answer and rendering it at once, the model emits tokens as produced and the interface renders them as they arrive. Total time does not change; perceived time collapses.

This is not a cosmetic trick. Perceived latency is the latency that churns users, and the streaming decision is among the cheapest product improvements available — no tokens, no model changes, no infrastructure.

First-token time is the span between the request leaving and the first piece of the answer arriving. It is dominated by what happens before generation starts: the queue, the prefill of the prompt, the round trip. Total time adds the generation itself — the long tail of tokens streaming out.

The perception asymmetry: a user will forgive a long generation that started quickly, and will not forgive a long silence even when the total is short. First-token time is the silence; generation is the progress. The latency post’s three numbers — first token, per-token rate, total — map onto the experience in exactly this order of importance.

Which is why the instrumentation advice from that post matters here: perceived latency cannot be improved before it is measured, and the first-token distribution is the measurement that predicts the feeling.

Three things — and none of them is the model. The transport: instead of one response at the end, the API delivers a stream of chunks, each carrying a few tokens, rendered incrementally by the client. The rendering: the interface must be built to append — a chat bubble that grows, a table that fills — rather than to swap in a finished block. And the failure handling: a stream can end early, the truncation case the resilience post covers, and the client must tell a finished answer from a dropped connection.

Streaming also changes the cancellation economics. A user who stops reading can stop the generation — the request is cancelled, the remaining tokens never produced, and the bill stops where the attention stopped. Non-streaming systems pay for the whole answer whether anyone read it or not.

None of this requires exotic engineering. The API surface is standard, the client work is a rendering loop, and the payoff is the largest perceived-latency win available for the cost.

When the consumer is not a person at all. A pipeline that parses the answer needs the whole answer before it can begin — streaming to a parser is delivery in pieces, and the pieces add complexity without adding value. Batch workloads, eval runs, anything whose deadline is measured in hours: the batch post rule applies, and nobody is watching the spinner.

Streaming also hurts when the interface is not built for it. A UI re-rendering the whole block on every chunk flickers and stutters; a badly rendered stream is worse than no stream at all. The rendering loop is the product, not the transport.

And it hurts when the answer is short. A one-line response arrives in a single chunk either way, and the streaming machinery is pure overhead. The decision is per-surface, not per-product.

The interface work is small and specific. Draw the first chunk the moment it arrives — no waiting for a sentence boundary or a minimum length; the first token is the psychological event. Show the thinking: a cursor, a typing indicator, the model’s name — anything converting waiting into watching. And keep the layout stable: reserve the space the answer will occupy, so the page does not jump as the text grows.

On the measurement side, instrument the two numbers that matter to the feeling: time to first visible token, and the gap between chunks. A stream stalling for seconds between chunks feels broken even when the average is fine — the tail discipline from the latency post applies to chunk gaps too.

And respect the reader’s time: the cancellation path is a feature. A user who can stop a wrong answer early is a user who stays for the next question.

Not directly — perception is not a clock. What can be measured is the proxy that predicts it: first-token time at the percentiles that matter, chunk-gap stalls, and abandonment. Abandonment is the honest metric: how often users leave before the answer finishes, plotted against first-token time. When the curve bends, that is the number the users feel.

The instrumentation is the same three timestamps from the latency post, plus one: the moment the first chunk was rendered, not just received. The gap between received and rendered is your own code, and it is usually the cheapest milliseconds available.

Measure, then stream, then measure again. The order matters — streaming is the fix for a specific feeling, and the measurement tells whether the feeling moved.