LLM Inference Engineering
The engineering side of serving models in production: model routing, prompt caching, API migration, and vendor evaluation. Written for the people who own the endpoint when it pages.
Structured Outputs: JSON You Can Actually Parse
The most common LLM integration failure is the answer that will not parse: JSON mode, schema enforcement, and the validation gate that makes output trustworthy.
Prompt Injection: The Threat Nobody Budgeted For
Prompt injection needs no exploit — just a sentence in the input: the threat model, the defense layers, and why the tool list is the blast radius.
What Is a Token, Anyway?
What a token actually is: how the tokenizer cuts text, why language and format change the count, and how to measure instead of guess.
Streaming: Why First-Token Time Beats Total Time
Users forgive long answers that start fast and never forgive silence: first-token time, streaming mechanics, and the rendering loop that changes the feeling.
Semantic Caching: When the Answer Already Exists
Semantic caching returns stored answers for new questions that mean the same thing: when it pays, the false-hit problem, and the invalidation nobody demos.
The Catalog Moves: Living With Model Versions and Deprecations
Model catalogs move: versions ship, prices change, old ids retire. Pin vs float, what a healthy deprecation policy looks like, and how to never miss one.
Timeouts, Retries, Idempotency: The Resilience Checklist Nobody Writes Down
Every LLM integration fails the same five ways. The unglamorous checklist — timeouts, retries, idempotency, circuit breakers, degradation — in one place.
Latency Is a Feature You Pay For
LLM latency is three numbers, not one: time to first token, per-token speed, total time. What each is ruled by, and which knobs you actually control.
Rate Limits Are an Architecture Input, Not an Error
A 429 is not an exception, it is the contract. Reading rate limits as a capacity plan, the four patterns that absorb them, and what retry storms cost.
The Enterprise Inference Vendor Checklist
A procurement-ready checklist for evaluating LLM inference providers: data policy, pricing transparency, reliability evidence, catalog, and exit terms.
Prompt Caching and the Economics of the Static Prefix
The static prefix is the quiet multiplier on the bill: how prompt caching works, the prefix rule that earns the discount, and the five things that poison it.
Why Hard-Coding One Model Is Now Your Biggest AI Cost Line
Pinning every request to one LLM feels simple. It quietly sets your price ceiling, your quality floor, and your negotiating position.
What "OpenAI-Compatible" Actually Buys You
Every inference provider claims an OpenAI-compatible API. What compatibility actually covers, what it leaves behind, and how to test it in an afternoon.
Embeddings and Vector Search: The Hidden Cost
Embeddings and vector search sit between question and answer — and their cost is rarely priced right. The three bills, the index lifecycle, and the quality dials.
LLM Observability: What Logs Miss
Logs show what happened; traces show why. The observability stack that catches drift, the dashboards that earn their noise, and what to watch first.
Quantization: Smaller Models, Cheaper Bills
Quantization shrinks models, speeds them up, cuts the bill — at a quality price. The mechanism, the eval discipline, and when to leave the weights alone.
Context Window Management Without Losing the Thread
Every conversation eventually outgrows the context window. Summarization, retrieval, and the other strategies — and what each one costs in fidelity.
Browse other topics