Fine-Tune, Prompt, or RAG? A Decision That Deserves Better Than a Default

Every team that outgrows a stock model meets the same three-way fork, and most take whichever branch they heard about first. The levers are not interchangeable, and the confusion between them is expensive — so the cleanest framing: prompting tells the model what to do right now, retrieval tells the model what is true right now, and fine-tuning changes what the model is.

Prompt engineering is free to try and instant to iterate. Retrieval-augmented generation bolts a search step onto the prompt, so the model answers from your documents instead of its training data. Fine-tuning trains the model itself on your examples, producing weights you own. The first two live entirely at request time; the third moves the work to training time and pays back on every subsequent request.

The decision is not “which is best” but “which problem do I actually have” — and most failed projects picked a lever that could not, in principle, solve their problem.

Prompting is the cheapest experiment in machine learning: no infrastructure, no pipeline, results in minutes. A well-structured prompt — clear instructions, a defined output format, a few worked examples — closes a surprising share of the gap between stock and custom. Teams routinely reach for training when a tighter prompt would have done it, and then maintain a training pipeline for the rest of the year.

The prompt is also where the real problem gets diagnosed. Watch how a stock model fails the task. Better instructions and examples fix it? There was never a training problem. The model knows nothing about the domain’s facts? That is a knowledge gap, and no instruction adds facts — retrieval’s job. The model knows the facts and understands the instructions but cannot hold the format, the voice, or the judgment consistently? That is a behaviour problem, and behaviour at scale is fine-tuning’s job.

Treat the prompting phase as diagnosis, not just attempt. Its failures are the specification for whatever is built next.

Retrieval earns its place when the answer depends on proprietary knowledge, recent, or both: the documentation, the tickets, the policies, the inventory. The model cannot know these things — they were not in its training data — and retraining every time a document changes is a category error. Retrieval keeps knowledge outside the model, updatable by editing a document instead of running a training job.

The costs are real and should be budgeted: a retrieval pipeline to build and maintain, chunks of retrieved text inflating every prompt, and a new failure mode — the right answer exists in the corpus but the retriever did not find it. The prompt-size side has its own economics, covered in prompt caching and the static prefix, and the stuff-versus-retrieve crossover gets its own treatment in long context vs RAG.

What retrieval cannot do is behaviour. A model quoting the documents in the wrong voice, answering in the wrong structure, missing the judgment calls the experts make — no retriever fixes that. Knowledge is retrieved; behaviour is trained.

Fine-tuning is the answer when the model itself needs to be different: a house style surviving any prompt, an output format that never drifts, a classification judgment tuned to the edge cases, a small model performing like a bigger one on a narrow task. Training on examples moves behaviour from the prompt — consuming tokens on every request, varying with every phrasing — into the weights, where it is stable and free at request time.

The economics matter at scale. A long system prompt full of style rules and examples is billed on every call, forever; a fine-tuned model carries the same behaviour for nothing per request. Past some volume the training run pays for itself — and the result is an asset owned outright, not a prompt rented. On Vidman AI the trained weights are yours, the difference between customising a model and depending on one; the mechanics are in the SFT guide and the training docs.

What fine-tuning cannot do is add living knowledge. A fine-tuned model’s facts freeze at training time, and asking training to keep up with a changing knowledge base is the classic misuse. Facts that change belong in retrieval; behaviour that must hold belongs in the weights.

Each lever carries a characteristic billing signature, and matching it to the traffic shape is most of the decision. Prompting bills forever: instructions, examples, style rules ride every request, so cost scales linearly with volume and never amortises. A long, beautiful system prompt is a recurring subscription paid per token — which is why the prompt caching post treats the static prefix as an asset worth engineering.

Retrieval bills per request plus a standing cost: smaller prompts, but an index to build, refresh, and operate. Its curve starts above zero and rises slowly; prompting starts at zero and rises steeply.

Fine-tuning inverts the shape: real upfront cost — the training run, the data work, the evaluation — then requests both cheaper and shorter, because the behaviour lives in the weights instead of the prompt. The crossover against a fat system prompt is arithmetic runnable on your own volumes — the same break-even logic as serverless vs dedicated applied one layer up.

The pattern to recognise in the numbers: high volume plus a long prompt is the signature of a workload begging to be fine-tuned; low volume plus a fast-moving knowledge base is retrieval’s home turf; everything in the prototyping phase belongs to the prompt.

The “pick one” framing survives almost no contact with a real product. The common production shape is a fine-tuned model behind a retrieval step: the retriever supplies the facts of the moment, the trained weights supply voice, format, and judgment, and a thin prompt supplies per-request instructions. Each lever does the job it is good at, and none is asked to impersonate another.

There is also a useful sequence. Teams that fine-tune first often discover they built the wrong thing; teams that prompt first arrive at fine-tuning with a precise specification of the behaviour gap and a folder of real examples — which then becomes the training set. Sizing that set is its own question, taken up in how much data do you need to fine-tune.

When none of the three gaps exists: the model already knows the facts, already holds the behaviour, and the prompt already fits — a stock model is the answer, and the discipline is accepting it. The serving-modes post argues the same from the infrastructure side: measure on stock models, and only add machinery when the measurement names a gap.

Stock is also right while the workload is still finding its shape. A product whose requirements move weekly cannot be trained — the dataset would chase it — and a retrieval index built for last month’s documents is already wrong. The default until the shape settles is the strongest stock model plus the best prompt.

And the honest check is the eval from the eval-before-purchase post: run the stock model against the acceptance bar before concluding anything is missing. A surprising share of “we need to customise” decisions die the day the stock model is actually measured.