The Eval Comes Before the Purchase

The usual way teams pick a model takes under an hour: open a public leaderboard, sort by the headline score, shortlist the top few, integrate the one that fits the budget. Fast, defensible in a meeting — and answering a question you do not have.

A leaderboard measures performance on a fixed public benchmark: exam questions, math problems, crowd-sourced preference votes. The product does not answer exam questions. It classifies tickets, extracts fields, drafts in a voice, reasons over documents. The gap between benchmark performance and workload performance is not a rounding error; it is routinely wide enough to flip rankings. A model dominating the aggregate board can lose on your task to a smaller model that happens to be strong at exactly your shape of work.

This is the first post in this cluster that asks for effort before money: build the evaluation first, buy second. The arithmetic of what a passing model costs is covered in cost per completed task; this post is about the scoreboard that arithmetic depends on.

The raw material is already in the logs. Pull a slice of real production requests — a few hundred to start, more if the workload is diverse — and strip anything sensitive. The goal is not a sample of easy cases but a faithful cross-section: the routine majority plus the long tail that actually breaks models. If traffic splits into distinct task shapes, sample each separately so one dominant case does not drown the rest.

For each example, record the input exactly as the model would see it and what a correct answer looks like. “Correct” does not have to mean one golden string. Extraction and classification can use exact match; drafting usually needs a rubric — the facts that must appear, the format that must hold, the tone that must not slip. Rubric-graded answers take longer to write down, and they are the difference between an eval and a demo.

Two disciplines pay off immediately. Version the set — it will be rerun for years, and reruns need to know what changed. And include the failures already known: the prompt that broke the last model is worth more than ten prompts that never broke anything.

An eval with no pass bar is just content generation. Before running a single model, write down what “good enough to ship” means, as a check a colleague could apply. Three families cover most workloads.

Correctness: does the answer contain the right facts, fields, or classification? This is where the exact-match and rubric work pays off.

Format discipline: is the output parseable by the system consuming it — valid JSON where promised, the right schema, no trailing prose? A brilliant but undisciplined model is expensive in a way per-token pricing never shows, because every malformed answer is a paid retry.

Refusal and edge behaviour: what does the model do with requests it should decline or cannot answer? A model confidently inventing an answer to an out-of-scope question fails differently — and worse — than one that says so.

Set the bar where the product needs it, not where it flatters a model. The bar is a product decision; the eval just enforces it.

Exact match grades itself; rubric answers need a judge — and that is where eval projects stall: someone realises a human must read hundreds of outputs, and the eval quietly never happens. The practical answer is a model grader: a strong model given the rubric, scoring each answer — with the crucial caveat that the judge is a model with a model’s failure modes. It has blind spots, it can be charmed by fluent wrongness, and it tends to grade its own family’s style generously.

The discipline that makes model grading trustworthy is the spot check. Humans grade a small slice of every run; if judge scores and human scores disagree beyond tolerance, the rubric or the judge gets fixed before anything ships. The judge exists to make grading cheap enough to run constantly, not to make human judgment unnecessary.

A second trap: letting the judge’s convenience redefine correctness. A rubric quietly rewritten into whatever the judge scores reliably measures measurability. Keep the rubric owned by the product side — the people who set the acceptance bar — and let the tooling serve it, not steer it.

Run the candidate models against the set, strict about the order of operations. Quality is a gate, price is a ranking: a model either clears the acceptance bar or it does not — and the ones that do not are not “cheaper options”, they are not options. This single discipline prevents the most common procurement error in the market: choosing a model for price, then spending the savings on retries, guardrails, and apologies.

Among the survivors, rank by cost per completed task: the all-in cost of an accepted answer, retries and wasted output included, not the sticker rate. The framework and its traps are in the cost-per-task post; the per-token inputs are on the pricing calculator, which reads the same published rates as the model library.

Notice what this does to the shortlist: it usually surfaces a model several tiers below the leaderboard’s suggestion. That is not the eval being lenient. That is the eval doing its job.

Most teams treat evaluation as a procurement step and discard it once the contract is signed — the expensive reading. The eval just built is a regression suite for every future decision: a new model releases and “should we switch” has a same-day answer; a provider changes a price or deprecates a version and the survivors re-rank instead of guessing; a prompt change is proposed and the eval says whether it helped before production does.

It also converts vendor claims into testable propositions. A provider announces a flagship; the press release says it is better; the eval says whether it is better at your work. The catalog will keep moving — the argument in how to read a price list for dating every comparison — and teams that re-evaluate in hours instead of quarters are the ones whose bills keep falling as the market improves.

Keep the set small enough to run cheaply and fresh enough to match live traffic. A stale eval drifts from the workload it was built from, and a drifted eval quietly re-becomes a leaderboard: accurate about everything except the product.

Low-stakes, low-volume workloads — an internal helper, a prototype, a tool a dozen colleagues use — make a structured eval overkill. Try the two or three obvious candidates on a handful of real prompts, pick the one that reads best, and spend the remaining effort on the product.

The upgrade trigger is the same as everywhere else on this blog: the moment someone outside the team depends on the output, the eval stops being ceremony. Paying customers, contractual accuracy, anything where a wrong answer has a cost — the acceptance bar earns its keep exactly then.

And the smallest version is never a waste: a dozen real prompts with written correct answers, graded by hand. It will not be statistically meaningful, and it will catch the obvious disasters — which, before anyone depends on the model, is the entire job.