Self-Hosting LLMs: The Full Cost

The self-hosting pitch always opens with the same slide: the GPU price divided by the tokens it can serve, set against the API’s per-token rate. The arithmetic shows the GPU winning by a wide margin, and the conclusion writes itself — why rent the tokens when you can own the machine?

The slide is not wrong; it is incomplete. The GPU is one line item in a system that includes power, cooling, networking, storage, redundancy, and the people who keep it running. A per-token comparison that counts only the hardware is the same category of error the price-list post warns about — a number describing no actual purchase.

This post is the full invoice: the line items the slide omits, the utilisation math that decides everything, and the honest comparison against the API.

Hardware: the GPUs, the servers around them, the storage for model weights and data. The facility: power, cooling, rack space — costs that run with every GPU hour, serving or idle. The network: bandwidth and connectivity that make the endpoint reachable. The redundancy: the second machine, the backup power, the failover path — a single GPU is a single point of failure, and production means the failure is planned for.

Then the people: the engineers who provision, patch, monitor, and fix. The people cost never appears on the slide and always appears on the invoice — and it does not scale down when the traffic does.

And the opportunity cost: capital tied up in hardware that could have gone to the product. The GPU is a depreciating asset; the product is the thing that compounds.

The items that surprise first-time self-hosters, in order of surprise. Idle time: the GPU bills every hour, traffic arrives in bursts, and the trough hours are pure loss. Overprovisioning: capacity must cover the peak, not the average, so part of the hardware is idle by design. Model updates: every new model version means new weights, new storage, new tuning — hardware that served last quarter’s model serves this quarter’s worse, and the upgrade cycle is a recurring capital expense.

Then the operational surprises: security patching, compliance questions, the weekend incident. The API provider absorbs these; the self-hoster staffs them.

And the quiet one: the exit. The hardware is bought, the model deployed, and the market moves — API prices fall, a new model arrives, and the self-hosted stack is now the expensive option, with sunk cost arguing against the switch.

The whole self-hosting economics collapse into one number: utilisation — the fraction of the GPU’s capacity actually serving tokens. A GPU at full utilisation is the slide’s arithmetic, and it wins. A GPU at a fraction is the slide’s arithmetic divided by that fraction, and it loses.

Utilisation is decided by the traffic pattern, not the engineering. A steady, predictable, always-on workload can keep a GPU hot; a bursty, spiky, seasonal workload cannot — the capacity covering the peak idles through the trough. The batch post’s argument about pay-per-use applies here with hardware: the spiky workload is exactly the one that should rent.

And the utilisation math includes maintenance windows, failed runs, redeploys — every hour the machine is not serving is an hour the slide assumed it was. The honest utilisation number is always lower than the planned one.

Three conditions, all necessary. Utilisation high and steady: the workload predictable enough to keep the hardware hot. Scale large enough to amortise the people: the engineering team spread across enough tokens that its cost per token is small. And requirements that demand it: data that cannot leave the building, latency that cannot tolerate a network, control the API cannot offer.

The third condition is the one that actually decides most self-hosting decisions — the requirements, not the arithmetic. Teams self-hosting for compliance or control are not comparing token prices; they are buying a property the API does not sell.

And the honest test is the same as every cost decision on this blog: model the full cost, people and idle time included, against the API at actual volume. The slide’s arithmetic is the starting point, never the conclusion.

When the workload is spiky, small, or still unproven. Bursty traffic pays for idle hardware; small traffic cannot amortise the people; unproven traffic is a bet made with capital instead of tokens. Pay-per-use exists precisely for these workloads, and the batch post’s rule applies: unproven workloads deserve pay-per-use.

It is also wrong when the model landscape is moving fast. Hardware bought for this quarter’s model is wrong for next quarter’s — the API customer switches models with a config change while the self-hoster runs a procurement.

And it is wrong when the team is not staffed for operations. The self-hosted stack is a production system with all the production responsibilities — monitoring, patching, on-call. Buying the GPU without buying the operations is how the slide’s savings become the invoice’s surprise.