Rate Limits Are an Architecture Input, Not an Error

Most integrations meet the rate limit the same way: testing goes fine, launch goes well, then one busy morning the error rates spike and someone is paged for a response saying, politely, “you are doing this too fast.” The team treats it as an outage. It is not an outage. It is the provider’s capacity plan arriving in the error logs, exactly as published.

Rate limits are the terms under which shared infrastructure stays shared: so many requests per minute, so many tokens per minute, sometimes so many concurrent generations. They are not hidden — they are in the docs and often in the response headers of every successful call. The failure is architectural: the limits were never read as an input to the design, so the design discovered them in production.

The fix is not a better try-except. It is treating the limit as a budget built around — the same way a database connection ceiling is.

Start by learning which currency the limit is actually in. Requests-per-minute ceilings bind chatty workloads with small prompts; tokens-per-minute ceilings bind workloads with long prompts or long answers, regardless of request count; concurrency limits bind workloads with long-running generations. A team can sit nowhere near its request limit while sitting on its token limit — and dashboards counting only requests will swear everything is fine.

Then do the arithmetic the other direction: take the peak minute, not the average day. Average traffic never trips a limit; peaks do. A morning rush, a marketing campaign, a nightly batch window — that is the number the limit has to survive. A peak that does not fit the published ceiling means a conversation with the provider before launch, not a retry loop after it — raised limits against committed volume are a normal enterprise discussion, and the questions to ask are in the vendor checklist.

Finally, keep the response headers. Most providers return the remaining budget on every call; logging it turns rate-limit management from incident response into a gauge on a dashboard.

A client-side queue. Your own system accepts work at whatever rate it arrives and releases it to the provider at the rate the contract allows — converting bursts into smooth flow, which is often the entire problem. A queue also forces the question of which work is actually urgent: the same question that unlocks batch pricing.

Backoff with jitter. When a 429 does arrive, retry after a delay that grows with each attempt, randomised so a hundred rejected clients do not all retry in the same second. Without jitter there is no retry policy — there is a synchronised hammer.

A concurrency cap. A hard ceiling on in-flight requests, set below the provider’s, so the system discovers its own throttle before the provider’s. Your own limit is one you can tune in advance; theirs arrives as an error.

Priorities. When capacity is tight, interactive requests should jump the queue and deferrable ones should wait. If a nightly report can delay an interactive chat answer, the queue is missing a lane.

The part that belongs in a cost discussion: naive retries do not just fail harder, they cost more. A request that fails after consuming its prompt still ran the prefill; a client retrying immediately, at full length, in a loop, converts a capacity problem into an invoice. Under sustained pressure, retry traffic competes with real traffic for the same limited budget, producing more 429s, producing more retries. Teams have watched the majority of spend during an incident go to requests that never produced an answer.

The defenses are the patterns above plus two disciplines. Idempotency: if a request might have succeeded before the connection dropped, the retry must not bill as a second execution of the same work — use idempotency keys where the API supports them. And a circuit breaker: past some failure threshold, stop retrying, shed load, fail fast. A system degrading to a clean “try again shortly” is cheaper and more honest than one that melts down thoroughly. The rest of the failure-handling checklist — timeouts, idempotency, circuit breakers — is in the resilience post.

This is also where deferrable work earns its keep twice: batch traffic can simply wait out the incident, because its deadline has room in it.

A team that has internalised all of this has a short, boring list on a dashboard. Queue depth: how much work is waiting for the provider right now, and whether it drains. Rejection rate: 429s per hour, plotted as a line that should sit at zero and means something specific when it does not. Remaining budget: the response-header counters, graphed, so an approaching ceiling is visible minutes before it is felt. And retry counts as a first-class metric — retries are the earliest symptom of every failure mode in this post.

With those four in place, the operational posture changes. Rate limits stop being surprises and start being seasonality: the shape of the peaks is learned, headroom shrinkage is watched months before it runs out, and the provider conversation happens while it is still a planning discussion. The unmonitored version of the same system finds out from a pager.

None of this requires exotic tooling. It requires deciding that the limits are telemetry — and treating that decision as seriously as the integration itself.

Sometimes the honest finding is that the architecture is chattier than the product requires. Agent loops calling the model for every micro-decision, polling patterns re-asking the same question, prompts re-sending a growing conversation in full when a summary would do — these multiply requests without multiplying value, and rate limits are simply where the multiplication shows up first.

Before asking for a higher ceiling, audit the request graph: how many model calls does one user action cause, and which could be cached, merged, or deleted? Cutting request count is the only rate-limit fix that also cuts the bill.

And when the workload is legitimately large and steady, the ceiling question becomes a capacity question — the point where shared infrastructure stops fitting at any limit is the break-even conversation in serverless vs dedicated. Rate limits are not a wall to resent; they are the shared tier telling you the truth about your size.

When you are the only user: a prototype, an internal tool with a dozen colleagues, a side project — read the limits once, set a retry with backoff and a cap, and stop. The queueing theory and the dashboards pay for themselves when strangers depend on the thing.

It is also overkill when the peak sits an order of magnitude under the published ceiling and the growth curve says it will stay there. Check the headroom once a month; spend the engineering on the product.

And the middle case — real users, modest headroom — deserves exactly the two cheap habits: log the budget headers, and set one alert on rejection rate. That is the smallest version of everything in this post, and it catches the incident before the pager does.