LLM Observability: What Logs Miss

A classic log line records that a request happened: endpoint, status, duration. For an LLM system that is the least interesting part of the story. The interesting parts are invisible to the log: what the model was asked, what it answered, whether the answer was any good, and whether today’s answer is worse than last week’s.

LLM systems fail in ways logs cannot see. The model drifts — same prompt, different quality, no error. The prompt changes and nobody notices the regression. The cost climbs and the invoice explains it a month late. The failures are slow, statistical, and silent.

Observability is the practice of making them visible: capturing the full request — prompt, response, tokens, latency, cost — and watching the aggregates move. The difference between finding out from a user and finding out from a dashboard.

The working unit of LLM observability is the trace: the complete record of one request’s journey — the prompt as sent, the response as received, token counts, latency breakdown, cost, and the metadata naming the model, the prompt version, the feature. A trace is a log line that remembered everything.

Traces compound into the questions that matter. Which prompt version is live? What did this user actually receive? What did this feature cost this week? The answers exist only if the traces were captured — retroactively, the data is gone.

And the capture is cheap relative to the model call it records: a trace costs a database write, the request costs tokens. That asymmetry is the entire argument — for a rounding error of the request’s cost, the request becomes inspectable forever.

Four numbers, each watched over time. Cost per completed task: the all-in spend divided by accepted answers — the metric catching every silent regression at once. Latency at the percentiles that matter: first-token and total distributions, because the average lies and the tail is what users feel. Error and retry rates: the failure taxonomy from the resilience discipline, because a rising retry rate is a falling margin. And quality: eval scores run continuously against live traffic, because the model can get cheaper and worse at the same time.

The dashboard’s job is not to show these numbers; it is to show their movement. A number is a fact; a trend is a signal. Teams that catch problems early watch the slopes, not the values.

And the dashboard should be boring. One demanding attention every day is noise; the good one sits quiet until a slope bends — and then it is the only thing anyone looks at.

Drift is the failure mode observability exists for: the system gets worse gradually, and no single request looks wrong. The model changes upstream and the answers shift; the prompt is edited and the tone moves; the traffic mix changes and the evals stop representing the users. Each change is small; the sum is a product that quietly stopped working.

The detection is the continuous eval: a slice of live traffic scored against the same rubric every day, plotted over time. When the curve bends, something drifted — and the trace data says what changed, because traces record the model version and prompt version of every request.

And the discipline is the response: drift findings get the same treatment as incidents. A drift alert acknowledged and ignored is a dashboard pretending to be a safety system. The alert exists to trigger the investigation, not to decorate the screen.

The alerting rule is the same as everywhere in operations: alert on things requiring action, silence the things that do not. An alert on every slow request is noise; an alert on the slow-request rate crossing a threshold is a signal. The threshold is the product decision, and it is written down.

The LLM-specific alerts are the ones logs cannot produce: cost per task crossing its budget, the eval score falling below its floor, the retry rate climbing past its ceiling — the alerts catching the failures this post opened with, the silent ones.

And every alert needs an owner and a runbook. An alert paging nobody is a decoration; one paging someone who does not know what to do is a punishment. The runbook is the difference between observability and anxiety.

A prototype with no users makes the full stack ceremony — the traces have nothing to record and the dashboards nothing to watch. The trigger is the same as everywhere on this blog: the moment someone else depends on the system, visibility stops being optional.

The overreaction to avoid is the opposite: instrumenting everything and watching nothing. A trace pipeline capturing every request into a database nobody queries is a cost with a dashboard attached. The stack earns its keep when the numbers are reviewed on a schedule and the alerts have owners.

And the honest sequence: capture the traces first, build the dashboard second, add the alerts third. Each step is worth building only once the previous one is being used. Observability is a habit, not a purchase.