The Cheapest Request Is the One That Can Wait

Teams optimising an inference bill reach for two levers: which model, and which provider. There is a third lever asked about far less: when. Not when in the cron sense — when in the deadline sense. How long can this specific request wait before its answer stops being useful?

That question is a pricing question, because urgency is what the real-time rate buys. A request answered in a heartbeat and a request answered by morning consume the same computation, but they are different products and deserve different price tiers. Providers know this: deferrable work fills idle capacity that would otherwise earn nothing, so work that can wait is worth a discount.

Most workloads were never asked the deadline question. They were built interactively because that was the default, and entire categories of deferrable traffic — reports, evaluations, backfills, enrichment — have been paying the urgency premium ever since. This post is about finding them.

Real-time traffic means a human or a downstream system actively waiting: chat interfaces, copilots, interactive search, agents mid-loop. The latency budget is part of the product here — a correct answer arriving after the user has tabbed away is not a correct answer, it is a contribution to a log.

This traffic earns its price tier. What is paid for is priority: the request jumps the queue, capacity is held ready, tail latency is engineered down because somebody’s attention is on the line. Saving money by degrading this half is borrowing against churn.

The one optimisation that belongs here is model choice, not scheduling: the smallest model clearing the quality bar, per request. Right-sizing a chat request saves money without touching its urgency. After that, leave the real-time path alone and go find the other half of the traffic.

The batch half covers everything whose deadline is measured in hours: the nightly report nobody opens before nine, the evaluation suite gating tomorrow’s deploy, the document backlog being enriched, the embeddings being rebuilt. This work is large, predictable, and patient — three properties providers prize, because patient work can be packed into the gaps interactive traffic leaves.

That packing is why batch tiers exist on price lists, and it is the same mechanism making below-list pricing possible in the first place — the utilisation economics are walked through in how GLM 5.2 is priced below list. Aggregation and batching are not marketing words; they are the physical reason deferred work is cheaper to serve, and the discount is the saving being shared.

The quiet scandal is how much batch-shaped traffic still rides the real-time tier. It happens innocently: the pipeline was prototyped against the interactive endpoint, nobody revisited the decision, and the cron job has been jumping the queue ever since.

Between “user staring at a spinner” and “read at nine tomorrow” sits a band of work easy to misclassify: the notification that should arrive within minutes, the review queue a human works through over a morning, the pipeline stage whose downstream step runs hourly. None need heartbeat latency, and none can wait all night.

The useful discipline is giving this band its own service-level statement instead of letting it inherit the real-time default. “Within minutes” is a designable target: a short queue, a modest polling interval, a worker that drains promptly. It costs a fraction of the urgency premium precisely because guaranteed immediacy has been replaced with an honest, bounded delay.

Human-in-the-loop workflows are the biggest residents of this band. If the output lands in a queue a reviewer checks between other tasks, the model’s speed was never the constraint — the reviewer’s afternoon was. Pricing such traffic as real-time is paying for a spinner nobody watches.

The test is a single question per workload: who or what is waiting, and what does a delay cost? A user staring at a spinner — real-time, conversation ends. Output consumed hours later — a dashboard refreshed nightly, a queue processed in the morning, a record nobody reads until audit season — the work is deferrable whether or not it is currently deferred.

The interesting cases sit in between. Some products have asynchronous-but-soon requirements: a notification that should arrive in minutes, a pipeline stage whose downstream step starts on the hour. Treat the deadline as a spectrum, and place each workload at the latest point on it the product honestly tolerates. Most teams discover their traffic sorts into a small urgent slice and a large patient mass — exactly the split the pricing tiers were built for.

Resist classifying by importance. Important and urgent are different axes: a compliance report can be the most important job of the night and still have all night to run.

Less dramatic than it sounds: same API, same models, two paths through your own system. Interactive requests go straight to the endpoint as they do today. Deferrable work goes through a queue — a table, a topic, whatever already runs — drained by a worker submitting on a schedule and writing results back where the morning expects them.

The queue earns its keep beyond pricing: it absorbs retries gracefully, smooths the rate-limit footprint (the four absorption patterns are in the rate-limit post), and turns “the provider had a bad hour in the night” from a page into a non-event. Deferred work is naturally resilient work, because the deadline has room in it.

One practical note on capacity shape: a patient mass growing large and steady enough turns the question from batch-versus-real-time into serverless-versus-dedicated — that crossover has its own arithmetic in the break-even post. The deployments docs cover both paths. Price the actual split with the calculator before committing to either.

Trap one is misclassification in the optimistic direction: deciding something is deferrable when it is not. The test is the consumer, not the producer — if any path from that output leads quickly back to a waiting human, it is interactive work wearing a batch costume, and the saving will be paid back in abandoned sessions.

The second is misclassification in the pessimistic direction — more common and more expensive: interactive defaults applied to work nobody waits on. Every internal tool, every back-office pipeline, every “temporary” script that became load-bearing tends to inherit the interactive tier from the day it was prototyped.

The third is operational: batch work fails silently unless made loud. A failed interactive request complains immediately — someone retries, someone notices. A failed batch job is discovered in the morning, sometimes. Deferred pipelines need their own alerting and retry discipline, because the whole point of deferring is that nobody is watching at the time.

When the product is genuinely and wholly interactive, there is no batch half to find — a chat product with no reporting surface should not invent one. The lever is model choice, and this post is not for you.

When the product quality is still being found, batch thinking is premature twice over. Development wants feedback loops short, not cheap — a slow, expensive, immediate answer beats a cheap overnight one when the loop is the point.

And when the deferral infrastructure is not worth building yet, the honest middle step is a tiny queue and a single worker — nothing exotic. The full split architecture above pays for itself once the patient mass is real; until then, a cron job and the interactive endpoint are a defensible v1.