Timeouts, Retries, Idempotency: The Resilience Checklist Nobody Writes Down

Model APIs are remote calls to a very large, very shared computer, and they inherit every distributed-systems failure mode plus a few of their own. The request that hangs forever. The response cut off mid-JSON. The timeout that fires after the model already produced the answer you are about to pay for twice. The burst of failures that turns out to be a rate limit. And the slow degradation — answers getting later and later — that never trips an error at all.

None of these are exotic, and none mean the provider is uniquely unreliable. They are what production traffic looks like, everywhere, on every vendor. What differs between teams is whether each failure was a design input or a surprise.

This post is the checklist that should have been handed out before the first launch. It is deliberately boring: nothing here is clever, and all of it is load-bearing. Rate limits got their own treatment in the rate-limit post; everything else fits on one page.

An HTTP client’s default timeout is written for a fast web service, not for a model that may legitimately think for a minute. Two consequences follow, and both bite. Too short, and successful requests get cancelled mid-flight — while the completion is still paid for, unread. Left at the default-infinity some clients ship, a wedged connection holds a worker forever, which is how one slow upstream becomes your outage.

The right value comes from your own latency data, not the provider’s marketing: take the tail of the real total-time distribution — the slowest requests the product tolerates, as argued in the latency post — and set the ceiling a notch above it. Interactive paths get an aggressive ceiling because a user is waiting; batch paths get a generous one because nobody is.

Then log cancellations separately from failures. A timeout is not a provider error; it is your own policy firing, and mixing the two makes both invisible.

The dangerous retry comes after an ambiguous failure: the connection dropped after the request arrived, and whether the model answered is unknowable. Retry naively and a completed generation runs twice — double the tokens, and double the effect if the call had downstream side effects. The fix is idempotency: send a stable key with the request so the provider recognises a duplicate and returns the first result instead of running it again. Where the API supports idempotency keys, use them on every mutating call; where it does not, make the retry safe on your side by checking whether the work already landed.

The retry loop mechanics are settled engineering: exponential backoff, full jitter, a hard cap on attempts. Details and anti-patterns are in the rate-limit post’s retry-storm section — the short version is that a retry policy without jitter and a cap turns a blip into a self-inflicted denial of service.

And classify before retrying. A request rejected for content will fail forever; a request rejected for capacity will succeed later. Retrying the first is pure spend.

Once failures cross a threshold — not a single bad request, but a run of them — the correct behaviour is to stop calling and start failing fast. That is the entire idea of a circuit breaker: an open circuit returns an immediate, honest error instead of queueing users behind a dead dependency. The error page is cheaper than the timeout, in latency and in tokens.

The better version degrades instead of failing. Queue the deferrable work for later — the batch lane from the cheapest request is the one that can wait absorbs provider incidents almost for free. Serve a cached or simplified answer where one exists. Fall back to a second model if the abstraction allows it — the deepest argument for the two-line-switch architecture in the migration post: an alternative already integrated is the only fallback that exists when it matters.

Whatever the fallback, decide it before the incident. “What does the product do when the model is down” is a product question, and the worst time to hold that meeting is during the outage.

The nastiest failure mode sends back a perfectly valid response that is quietly incomplete. The model hit the output cap, or the stream dropped near the end, and what arrives is most of an answer — which, to a parsing consumer, is a malformed document that looks like a bug in your code.

The API tells you, if you look: every completion carries a finish reason, and “length” means the answer was cut by the cap, not completed. Check it. A response that ended on length is not a successful response, and treating it as one ships truncated answers to users with a green status code attached.

For structured output, add a parse gate: the response is not done until it parses and validates against the schema. That single check converts a whole class of silent corruption into an ordinary, retryable failure — which the rest of this checklist already knows how to handle.

For a prototype with one user, nearly all of it. Wire a timeout so nothing hangs forever, log failures somewhere actually looked at, and build the product. The checklist pays for itself when other people depend on the thing.

The trigger points are boring and predictable: a paying customer, a scheduled job nobody watches, a second engineer who will get paged. Any one means the next failure will be somebody’s bad morning, and the morning is cheaper to prevent than to attend.

What is never overkill is the one-line version: every call has a timeout, every retry has a cap, every failure is logged with enough context to reconstruct it. That much fits in the first commit — the difference between debugging an incident and narrating one.