Semantic Caching: When the Answer Already Exists

“Caching” appears in two LLM contexts that have almost nothing in common, and conflating them costs money. Prompt caching, covered in its own post, stores the static prefix of a prompt so repeated requests do not re-pay for the same instructions. It is exact: the same bytes, recognised, billed once.

Semantic caching stores answers: a new question meaning the same thing as one already answered gets the stored answer, with no model call at all. It is approximate: not the same bytes, but the same meaning.

One saves the prefix; the other saves the whole call. The first is a billing optimisation; the second is an architectural one — and like most architectural optimisations, it is brilliant where it fits and dangerous where it does not.

Three things: the question, in a comparable form — usually an embedding, a vector representing meaning rather than text; the answer, exactly as served; and the metadata deciding whether the answer is still trustworthy — when it was generated, what it depended on, and how similar a new question must be for reuse.

The matching is the interesting part. A new question is embedded, compared against stored questions, and past a similarity threshold the stored answer is served. “How do I reset my password” and “I forgot my password, what do I do” are different strings and the same question — the cache is the thing that knows.

The threshold is the entire product decision. Loose, and the cache serves wrong answers confidently; tight, and it never fires. Everything else in this post is the engineering around that one dial.

When the traffic repeats itself semantically. Support products where the same questions arrive in different words, documentation assistants, FAQ-style workloads — these carry hit rates that make the cache the cheapest infrastructure in the stack, because the marginal cost of a hit is a vector comparison instead of a model call.

The arithmetic is the cost-per-task framework with a new term: the all-in cost of an accepted answer now includes the cache’s miss rate. A cache hitting often enough to pay for its own misses is free latency and free tokens; the crossover is measurable, and the measurement is the only way to know.

And the latency win is real even when the cost win is small: a hit returns in milliseconds — the perceived-latency argument from the latency post, delivered by architecture instead of rendering.

The failure mode that keeps a careful engineer awake: the cache serves an answer that is almost right and subtly wrong, and the user cannot tell. A threshold set too loose, a question similar in words and different in meaning, an answer correct last month and not now — the cache converts these into confident errors, and confident errors are the most expensive kind.

The defenses are the same ones the eval post prescribes for judges: sample the hits. Log what the cache served, review a slice of it, and measure how often a human would have wanted the model called instead. A cache is a model with one training example per question; it deserves the same scrutiny.

And the escape hatch: for questions where wrongness is expensive, the threshold tightens or the cache turns off. Caching is per-surface, not per-platform — the support bot may cache aggressively while the medical assistant does not cache at all.

Every cache demo shows the hit; none shows the day the underlying truth changes — the price update, the policy change, the new model release — and the cache keeps serving the old answer. Invalidation is the discipline of knowing what an answer depends on and expiring it when the dependency moves.

The cheap version is time: answers expire after a window, chosen per surface. The honest version is dependency: when the document an answer was built from changes, the answer dies with it. The honest version is more work — and it is the difference between a cache and a liability.

And the versioning post lesson applies: the model itself is a dependency. An answer generated by last quarter’s model is stale the day the model changes, and a cache that does not know which model produced its entries cannot know when they expired.

When the questions never repeat. A creative drafting tool, a coding assistant on novel problems, anything where every request is genuinely new — the cache is a database of misses, and the embedding infrastructure is pure overhead.

It is also wrong when the cost of a wrong answer is high and the cost of a model call is low. The cache saves tokens; it risks trust. Where trust is the product, tokens are the cheaper currency.

And it is premature before the traffic exists. A cache sized for questions not yet seen is speculation; the hit-rate data comes first, the cache second. Measure the repetition, then buy the infrastructure — the same order as every other purchase on this blog.