How Much Data Do You Need to Fine-Tune?

Every evaluation call asks it: how many examples do we need? The truthful answer — it depends on the task, the method, and the model — is unsatisfying, but it is not a shrug. The answer has a shape, and the shape is more useful than any number.

What the field keeps confirming: the count matters far less than what the set contains. A small dataset of excellent, representative examples routinely beats a much larger pile of inconsistent ones, because fine-tuning teaches behaviour, and behaviour comes from the pattern in the data, not the volume of it. Noise at scale is still noise — just expensive noise.

So no threshold below. Instead: the variables that decide yours, the point where adding data stops paying, and a way to find your own answer with experiments small enough to run this week.

The dominant variable is how narrow the task is. One output format, one classification scheme, one voice — a single well-defined behaviour — is a small job. A domain’s worth of judgment is a large one. The more the desired behaviour overlaps what the base model already does, the less data you are buying.

The method changes the scale too. Adapter methods like LoRA learn a compact set of new parameters and get remarkable mileage from modest datasets; full fine-tuning rewrites more of the model and wants correspondingly more evidence; continued pre-training, which teaches knowledge rather than behaviour, is the hungriest of all. The method ladder and its trade-offs are mapped in the adapter comparison and the SFT guide.

And consistency multiplies everything. Examples that agree with each other — same format, same judgment calls, same conventions — let the model find the pattern fast. Examples scraped from different eras, authors, and standards do not add up to one behaviour; they add up to an argument, and no dataset size fixes an argument.

Fine-tuning is learning by demonstration: the model is trained to produce more of whatever the dataset shows. That makes the dataset a specification and every example a line in it. Contradictory lines produce contradictory behaviour; sloppy lines produce confident sloppiness. This is why the dataset preparation guide spends most of its length on cleaning and curation — the preparation is not preprocessing, it is the training.

The practical consequence: before scaling a dataset up, read it. A review pass finds the mislabelled rows, the format drift, the examples teaching exactly the behaviour you are trying to remove. Cutting the worst slice of a dataset improves the model more reliably than adding a fresh slice, and costs less in every currency.

One heuristic to keep: a dataset you would be proud to publish as your company’s style guide is ready; a dataset you have never read is not, whatever its size.

The curve of return per example starts steep and flattens — the first examples teach the format and the voice, the next fill in common cases, and from there each additional batch adds rarer and rarer coverage. Past the knee of that curve, more of the same data mostly teaches the model what it already learned, while evaluation gets slower and training gets longer.

Find your own knee empirically: train on a small slice, evaluate on a held-out set, double the data, evaluate again. When the gain from doubling shrinks below what the extra training and curation cost, that is your answer — for this task, this method, this model. It will differ for the next task, which is why no blog post can hand it to you.

The experiments are deliberately cheap: short runs on small slices, billed per second on whatever GPU tier the method needs, so the search for the knee costs a fraction of the training run it informs.

Examples come from three places, in rough order of worth. Production logs are the gold standard: real inputs, and — where the product has corrections, edits, or thumbs — real evidence of what the right answer was. Teams sitting on months of logged traffic are usually much closer to a training set than they think; the work is curation, not creation.

Subject-matter experts writing examples comes second: slower and pricier per example, and worth exactly that for the judgment cases logs cannot show. A few dozen expert examples of the hard cases often move a model more than a mountain of routine ones.

Synthetic data — a strong model generating examples — is third, with an honest warning attached: you are training your model to imitate another model, biases and blind spots included. It is excellent for volume around a pattern you have already verified, and poor as the source of the pattern itself. Generate with a model, filter with a rubric, and keep a human reading samples from every batch.

The dataset preparation guide covers formats, validation, and import; the sizing question and the sourcing question are one question seen from two sides.

Occasionally the true answer to “how much data” is “more than you have, and differently than you think” — because the gap is a knowledge gap, and fine-tuning is the wrong tool for it. If the model fails because it does not know your facts, retrieval is the fix; the full decision is walked through in fine-tune, prompt, or RAG.

“Not yet” is also the answer when the examples exist only as intentions. Teams regularly schedule a training run before the dataset exists, and the dataset work expands to fill months. Build the first small set by hand, train on it, and let a real model — however imperfect — tell you what the next thousand examples should look like.

And if the task is still moving, freeze the target before you count. A dataset sized for last month’s definition of the task is the wrong size by construction.