Latency Is a Feature You Pay For

Ask a room what “the latency” of a model is and you will get one number. The system does not have one number to give. A request’s timeline has three distinct segments, with different causes, different costs, different fixes.

Time to first token is everything before the answer starts: the queue in front of you, the prompt being read and processed, the scheduling onto hardware. It is dominated by input length and provider load — and it is what a user stares at before anything moves.

Per-token speed is the rhythm once the answer starts: how fast successive tokens appear. It is dominated by model size and the serving hardware, and it sets how long a long answer takes to finish.

Total time is the sum a wall clock sees — the only one the logs record by default. The practical point: “make it faster” is not a plan until you know which of the three is slow. A request can have a snappy first token and a glacial finish, or sit in a queue and then stream instantly — identical in a total-time average, opposite problems.

The asymmetry that surprises everyone: reading a prompt is parallel, writing the answer is not. The model processes the entire input at once — the prefill — but produces output one token at a time, each depending on every token before it. No shortcut, no parallelism to buy: a long answer takes its tokens times the per-token speed, full stop.

Two consequences. First, output length is the strongest latency lever owned. A prompt inviting the model to ramble costs wall-clock time exactly as surely as it costs tokens — they are the same resource. Tighter instructions about answer shape buy both. Second, this is why verbose models feel slow even on fast hardware: the per-token rate can be excellent and the total poor if the model uses four hundred tokens where forty would do.

Budgeting latency for a feature starts from the answer length the product actually needs — design prompts and output caps around it before reaching for anything more exotic.

Streaming changes when tokens reach the user, not when the model produces them. Total generation time is identical either way; what changes is that the user watches the answer arrive instead of staring at a spinner for its full duration. Perceived latency collapses; actual latency does not move.

That is not a trick — it is the correct engineering trade: human patience is spent mostly on silence, not on duration. An answer beginning immediately and taking a while reads as fast; the same answer delivered all at once after the same total reads as broken. Any interface with a human on the other end should stream.

The machine-facing case is the opposite. A pipeline consuming the full answer gains nothing from streaming and pays a small complexity cost — partial responses, reconnection logic, incomplete JSON. Stream for humans, buffer for machines, and keep the two defaults out of each other’s lanes.

Model size is the big lever: smaller models decode faster on the same hardware — the latency half of the right-sizing argument in the hidden cost of hard-coding one model. An answer that does not need frontier strength does not need frontier decode time.

Output length, from the previous section, is the second knob and the cheapest to turn.

The third is your position in the provider’s day. Shared capacity has rush hours; a request queued behind a burst waits before its first token regardless of the model chosen. Anything deferrable moved off the peak is free latency for the traffic that stays — the scheduling version of the argument in the cheapest request is the one that can wait.

The fourth is geography: the round trip to the endpoint is part of first-token time, and no provider beats the speed of light. Users and inference on different continents means some “model latency” is actually a plane ticket.

Averages conceal the experience that churns users. A feature whose median request feels instant but whose one-in-a-hundred hangs for ages is, for one percent of sessions, a broken product — and those sessions are disproportionately the long prompts and hard questions: the most engaged users.

So instrument the distribution, not the mean: the median for the typical feel, the high percentiles for the worst case the product tolerates. Then set the budget at the tail. “The median looks fine” is not a latency story.

The tail also tells you where to spend. A tail dominated by queueing points at capacity and scheduling; by long generations, at prompt and output design; by specific request shapes, at those shapes. Same symptom, three different purchases — the measurement is what stops you from buying the wrong fix.

Instrumentation here costs less than most teams expect, because the three numbers fall out of timestamps already capturable. Log the moment the request left, the moment the first token arrived, and the moment the response closed — for every call, tagged with model and prompt and completion sizes. First-token time, decode duration, and total time come out of those three stamps, segmented by whatever dimension is suspected: model, feature, prompt length, hour of day.

A week of that log usually settles the argument that started the investigation. The first-token curve by hour shows whether you are queueing behind other people’s peaks. Decode time versus completion length shows whether the model or the verbosity is at fault. And the tail, sliced by request shape, shows which feature owns the bad experiences.

One measurement warning: do not benchmark from a laptop on one prompt and call it the model’s speed. Cold calls, single samples, synthetic prompts produce numbers about the test, not about the service. Latency is a distribution over real traffic — measure it there or not at all.

If nobody waits on the output, latency is a cost centre pretending to be a metric. Batch pipelines, nightly jobs, evaluation suites — deadlines measured in hours, and paying for their speed buys a spinner nobody watches, as argued in the batch post.

It is also the wrong focus when the real problem is correctness. A fast wrong answer is not a latency success, and teams routinely polish response times while the acceptance rate quietly bleeds. Speed is a multiplier on value, not a substitute for it.

And it is premature before the three numbers exist. Measure first token, per-token rate, and total separately; find the segment actually slow; then spend. Latency work before that measurement is tourism.