Distillation: Teaching Small Models From Big Ones

Distillation is the oldest trick in the compression playbook, and it works on language models the way it worked on everything else: a large, capable model — the teacher — generates outputs, and a small model — the student — is trained to reproduce them. The student never sees the teacher’s weights; it sees the teacher’s behaviour, and it learns to imitate it.

The result is a small model carrying a surprising fraction of the large model’s capability at a fraction of the serving cost. The student is cheaper to run, faster to answer, easier to deploy — and it has absorbed the teacher’s patterns, not just its answers.

The catch is the same as every compression: the transfer is lossy. The student learns what the teacher’s outputs reveal, and the outputs do not reveal everything. The art of distillation is knowing what transfers and what does not.

One model generates, the other trains. The training data is the teacher’s outputs: prompts go to the teacher, responses are collected, and the student is fine-tuned on those prompt-response pairs. The student learns the teacher’s answers — and, more importantly, the teacher’s style of answering: the reasoning steps, the formatting, the tone.

The richer variants transfer more than the final answer. Instead of only the teacher’s chosen response, the student trains on the teacher’s full output distribution — the probabilities assigned to every candidate token. That distribution carries the teacher’s uncertainty and near-misses, a deeper signal than any single answer.

And the modern variant adds the reasoning trace: the teacher’s step-by-step thinking goes into the training data, so the student learns not just what the teacher concluded but how it got there. The reasoning is often the most valuable thing that transfers.

Transfers well: style, formatting, reasoning patterns, and the teacher’s behaviour on the tasks represented in the training data. The student picks up the teacher’s voice and its approach — the things that show up in the outputs.

What transfers poorly: the teacher’s depth of knowledge beyond the training distribution. The student learns the answers it was shown; it does not inherit the ability to reason about things it was never shown. A distilled model is brilliant on the covered tasks and ordinary everywhere else — the coverage of the distillation data is the ceiling of the student.

And what does not transfer at all: the teacher’s uncertainty calibration, refusal boundaries, safety behaviour — unless deliberately included in the data. A student trained only on confident answers becomes confidently wrong; a student trained without refusal examples never learns to refuse.

The quality of the distillation is the quality of the teacher’s outputs, and the volume is the breadth of the student. The data question is both: enough outputs to cover the tasks the student must handle, and good enough outputs that the student learns the right patterns.

The generation cost is the tuition: every training example is a teacher call, and a broad distillation is a large teacher bill. The economics are the same shape as every training decision on this blog — tuition paid once, the student’s savings compounding forever, the crossover is arithmetic.

And the data must include the failures. A student trained only on the teacher’s successes never learns what the teacher refuses, where it hesitates, when it says “I cannot”. The failure examples are where safety behaviour lives, and omitting them is how distilled models earn their reputation for overconfidence.

When the task is narrow enough that the teacher’s breadth is wasted. A student distilled for one product’s tasks — the support answers, the classification, the extraction — can match the teacher on those tasks at a fraction of the cost, because it does not need the teacher’s everything-else.

It also wins on the operational axes: latency, deployment footprint, serving cost. The distilled model runs where the teacher cannot — on the edge, in the tight loop, at high volume. The capability tradeoff buys the operational fit.

And the honest comparison is the eval: the student against the teacher on the product’s actual tasks. The eval decides whether the compression was worth it — the same discipline as every model choice on this blog, with the teacher as the control group.

When the task needs the teacher’s full depth. Open-ended reasoning, novel problems, anything where the training distribution cannot cover real traffic — the student’s ceiling is the data, and the data cannot cover what has not happened yet.

It is also wrong when the teacher itself is the product. If the differentiation is frontier capability, the distilled student is a cheaper imitation of the thing being sold — the savings come out of the product.

And it is wrong when the tuition exceeds the savings. A distillation demanding an enormous teacher bill to cover a small task set is a purchase, not an investment — the crossover math decides, and it is run before generation starts, not after.