Open vs Closed Models in Production: Cost per Completed Task, Not Cost per Token

Every model comparison online ranks by the same number: price per token. Easy to publish, easy to sort — and it answers a question nobody actually has. You do not buy tokens. You buy completed tasks: resolved tickets, extracted fields, drafted replies, working code. The token is just the unit the bill is printed in.

Between token price and task price stands everything a model can do wrong: ramble, refuse, emit broken JSON, fail a validation and get retried, produce an answer a human has to redo. Each consumes paid-for tokens and contributes nothing to a completed task. Two models can sit far apart on a per-token table and land in the opposite order once the finished job is priced.

This post is a framework, not a leaderboard. No table of the catalog ranked by cost per task will be published — that benchmark has not been run, and a ranking without a benchmark behind it is exactly the content this article argues against. The method is what is offered, so the number acted on is measured on your workload instead of invented here — the how-to is in the eval-before-purchase post.

Verbosity is the first leak in the arithmetic. Models differ enormously in how many tokens they spend saying the same thing — reasoning traces, restatements, hedging, courtesy padding. Output tokens are usually the expensive half of any rate card, so a model that talks more bills more, even when the extra words add nothing. A cheap-per-token model with a long-winded style can cost more per answer than a terse premium one.

Retries are the second leak. A model failing the format check or the quality bar on some share of requests gets paid for the failures, then paid again for the recovery. Its effective price is the token price divided by its success rate on your tasks — and the success rate is the part no pricing page can tell you.

The third leak is subtle: the cheapest model that cannot do the task at all has an infinite cost per completed task. Teams discover this when they route work down-market to save money and end up building validation, repair prompts, and human escalation paths around the gap. The engineering around a too-weak model is part of its price.

Defined precisely, it stops being a slogan. Take a fixed set of real tasks from the product. Run each candidate model over the whole set. A task counts as complete only when its output passes the acceptance check — schema-valid, correct, shippable, whatever the bar is. The model’s cost per completed task is everything billed across the run — first attempts, retries, repair prompts, all of it — divided by the number of tasks that passed.

Written as a loop, the measurement is deliberately unglamorous:

for task in tasks:                    # your real prompts, frozen
    attempts = 0
    cost = 0
    while True:
        attempts += 1
        result = call_model(model, task)
        cost += result.billed_cost    # every attempt counts
        if passes_acceptance(result): # your bar, not a benchmark's
            break
        if attempts > max_attempts:   # a failure is a cost too
            break
    record(model, task, cost, passed)

The details that make it honest are all in the discipline: the task set is frozen before starting, the acceptance check is written before any outputs are seen, and failures count at full cost. Change any of those after looking at results and it is an advertisement, not a measurement.

Build the task set from production traffic, not from a benchmark suite. Sample real requests across the actual mix of work the product does — including the boring majority, because that is where the money is. Weighting matters: a model shining on the hardest showcase task but drowning on the everyday volume is the wrong buy.

Freeze everything before the first call: the tasks, the prompts, the acceptance checks, the retry policy. Then run every candidate against the identical rig. The temptation to tune a prompt mid-run for the model expected to win is strong, and it converts the exercise from measurement into advocacy. A prompt needing to change means the run restarts.

Keep the raw outputs, not just the scores. When the ranking surprises — and it will — the artifacts are how you find out whether you measured the models or a bug in the harness.

The ranking table inverts more often than anyone expects. Terse, well-behaved models climb; eloquent ones sink. Models following output schemas reliably beat smarter models that freelance on format, because a failed parse is a retry, and a retry is a full-price second attempt.

The open-versus-closed question changes shape too. Per-token tables make it a simple price ordering; per-task measurement makes it a question about task classes. Open models are strong — often excellent — on the structured work making up most production traffic: classification, extraction, transformation, grounded question-answering. The remaining gap concentrates in open-ended, ambiguous, or genuinely novel reasoning. Which share of the workload that is decides the answer, and the share is unknowable until measured.

The catalog to run this against — open frontier models alongside the closed ones, all behind the same API shape — is on the model library, and the per-token side of the math is already done on the pricing calculator. The task-cost side is the part only you can supply.

Cost per completed task is one axis; treating it as the only axis is how teams buy cheap chaos. The measurement means nothing below a quality floor set in advance: the acceptance check defines “complete”, and a bar set where users would be disappointed prices work you would never ship. Floor first, then minimise cost above it — never the other way round.

Latency belongs in the same frame. A model cheap per task but slow enough to hurt the experience is not cheap; it is subsidised by your users’ patience. Interactive products should bound acceptable latency before reading the ranking, and batch pipelines should notice their constraint runs the other way — throughput per dollar, not milliseconds per request.

The honest summary is three numbers per model, not one: what a completed task costs, how often it clears the quality bar, and how long it takes. Anything simpler is a lobby poster.

When the task is genuinely novel, yes. Frontier closed models earn their rate on work with no template: ambiguous specifications, long-horizon reasoning, problems where the cost of a wrong answer dwarfs the cost of the tokens. If a failure means a lost customer or a bad headline, the premium model is the cheap option.

When the volume is small. A task running a handful of times a day makes the difference between a good model and the best model pocket change, and the measurement exercise costs more than it will ever save. Spend the rigor where the volume is.

And when the acceptance check cannot be automated, per-task costing still works — it just needs a human in the loop, which makes it slower and more expensive to run. That is an argument for keeping the set small, not for skipping the measurement: on high-stakes work, the human-graded version is the only version that counts.

Everything in this post is runnable this week. Pull a hundred real tasks, write the acceptance check, run two or three candidate models — the rig is smaller than the demos suggest, and the first run teaches more than any blog post, this one included.

The eval-before-purchase post carries the full apparatus: building the set from traffic, the acceptance bar, grading at scale. The arithmetic after that is the loop above. And the platform side is already open: the model library lists the catalog, the pricing calculator prices the tokens, and the same API shape serves every candidate, open or closed.

Start with the models already in use as the control group. The surprise is usually not that the ranking changes — it is how much.