Embeddings and Vector Search: The Hidden Cost

Nearly all of the cost conversation about LLM systems is about the model: tokens in, tokens out, the per-million rate. But systems that actually work in production carry a second layer that nobody prices correctly — the retrieval layer. The embeddings that turn text into vectors, the index that stores them, and the search that finds the relevant pieces before the model ever sees the question.

The layer is not optional for most products: it is how the model knows your documents, your products, your history. And it has its own cost structure — embedding calls billed per token, index storage billed per vector, search billed per query. Three bills where most teams budgeted for one.

The layer is also where quality lives or dies. The model can only answer from what retrieval finds; a retrieval layer returning the wrong passages produces a confident model answering the wrong question. The cost conversation and the quality conversation are one conversation.

A list of numbers representing the meaning of a piece of text. The embedding model reads a sentence and produces a vector — a point in a high-dimensional space — positioned so texts with similar meanings land near each other. “How do I reset my password” and “I forgot my password” produce nearby points; “password” and “pineapple” do not.

The search is the geometry: a new question is embedded, and the index finds the stored vectors nearest to it. That is the entire retrieval trick — meaning converted to distance, distance computed by arithmetic.

And the embedding model is a model like any other: tokenizer, context window, per-token price. The cost discipline from the token post applies unchanged — the difference is that embedding calls are invisible, fired in bulk, and rarely counted.

Three separate bills, each with its own failure mode. The embedding bill: every document is embedded on entry, every query on arrival. The document side is a one-time cost per document; the query side is per-request forever. Teams that budgeted the query side and forgot ingestion discover the difference on the first full re-index.

The storage bill: every vector is stored, and the index grows with the corpus — per vector per month, compounding. A corpus growing steadily is a storage bill growing steadily, long after the ingestion budget was spent.

The search bill: every query costs a search, scaling with index size and the number of results requested. The model call that follows is the expensive part, but the search is the part that runs on every request — including the ones retrieving nothing useful.

And the hidden fourth: re-embedding. When the embedding model changes — and it will — the entire corpus must be re-embedded, and the one-time ingestion cost becomes a recurring event.

The index is the database of vectors, and it carries the same lifecycle costs as any database: storage, compute for search, and the operational work of keeping it consistent with the source documents. A document changes, the index must change with it — the update path is where retrieval systems quietly rot.

The consistency question is the expensive one: does the index update in real time, in batches, or on a schedule? Real-time updates cost the most and prevent the most embarrassing failure — an answer citing a document corrected last week. Batch updates are cheaper and guarantee that failure happens sometimes.

And the index has quality dials too: results retrieved per query, the similarity threshold, the reranking step. Each turn of a dial changes cost and quality together. The retrieval layer is a system with its own tuning surface, and the tuning is part of the job.

The cheap setup — small embedding model, loose threshold, no reranking — retrieves fast and retrieves wrong. The expensive setup — strong embedding model, a reranker on top, a tight threshold — retrieves right and bills accordingly. The quality of the model’s answers is bounded by this choice, and the choice is a budget decision nobody made consciously.

The reranker is the clearest example: a second model reading the retrieved candidates and ordering them by relevance. It doubles the retrieval cost and dramatically improves the answers. Teams that skip it save money on retrieval and spend it on model calls answering from the wrong passages.

The honest framing: retrieval quality is measured in the same currency as everything else on this blog — cost per completed task. A retrieval layer that finds the right passage first try is cheap at any price; one that finds it on the third retry is expensive at any discount.

When the corpus is small enough to fit the context window directly. A handful of documents, a short FAQ, a product catalog that fits the prompt — the retrieval layer is pure overhead, and the model can read everything itself. The crossover is arithmetic: when the corpus fits, retrieval is a tax.

It is also wrong when the queries are exact rather than semantic. A lookup by ID, a search for an exact string, a filter on a field — those are database queries, and a vector index is an expensive way to run them. The semantic layer earns its keep on meaning, not on matching.

And it is wrong when the corpus changes faster than the index can follow. Real-time data, rapidly edited documents, anything where staleness is a correctness bug — the retrieval layer becomes the source of the errors it was built to prevent. The index is only as good as its update path.