Full Fine-Tuning: When and Why to Train Every Parameter

Full fine-tuning is adaptation in its plainest form: every parameter updates. No frozen weights, no low-rank approximations, no adapters, no quantised base. Training operates over the complete parameter space, as pre-training did, but on your task data and at a much lower learning rate so the pre-trained knowledge survives.

Parameter-efficient methods like LoRA and QLoRA add small trainable pieces and freeze the rest; full fine-tuning hands the optimiser the whole network. The freedom buys representational change that adapters cannot reach — which is why large diverse datasets and deep domain adaptation are where the method earns its premium.

It is also the natural companion to continued pre-training: extending a model’s knowledge over a large unlabelled corpus uses the same next-token objective as the original pre-training, and low-rank updates are too restrictive a vehicle for genuinely new knowledge.

Capacity, first. LoRA and QLoRA confine updates to a low-dimensional subspace set by the rank; for instruction-following, that constraint rarely binds, but for heavy knowledge injection, large behavioural change, or multi-step reasoning adaptation it becomes the ceiling on final quality.

Deployment cleanliness, second. A LoRA model needs either a merge step or runtime adapter application; a full fine-tune is one set of weights, no extra components, inference performance identical to the base architecture.

And scale, third. The quality advantage over adapters grows with dataset size: below a few thousand examples the difference is negligible or even favours adapters (they overfit less); at tens of thousands, full fine-tuning pulls clearly ahead. That curve is one of the main inputs to the method decision.

Substantially more than adapters, because gradients and optimiser states exist for every parameter, not just small adapter matrices. With Adam and bfloat16 weights, budget roughly 16 to 20 bytes per parameter: 2 for weights, 2 for gradients, 4 for the first moment, 4 for the second — plus activations for the pass in flight.

Practically: a 7B model costs about 56 GB, a 13B about 104 GB, a 70B about 560 GB — past a single server, requiring distribution across nodes. Gradient checkpointing (recompute activations, save memory) and gradient accumulation (smaller per-device batches) both pull the number down.

Multi-device and multi-node coordination for large models is handled by the platform automatically — you set the model and the configuration, the training job distributes itself.

Four scenarios argue for it. Continued pre-training first: hundreds of millions or billions of tokens of specialised text — legal corpora, biomedical literature, a codebase — where the low-rank constraint cannot carry the knowledge being added.

Second, domain adaptation on a large, high-quality instruction set: past roughly 50,000 examples with a real behavioural shift, full fine-tuning beats LoRA meaningfully, because the extra capacity is what absorbs that much signal.

Third, absolute inference performance: no merge step, no runtime adapter — the fastest possible serving path for latency-sensitive scale.

Fourth, changes to the output distribution that low-rank updates struggle to reach: deep stylistic shifts, multilingual expansion, domain-specific terminology — the tasks where adapter quality plateaus and full fine-tuning does not.

For a 7B model, around 10,000 high-quality examples before full fine-tuning pulls meaningfully ahead of LoRA; below that, the low-rank constraint’s regularisation helps more than it hurts, while full fine-tuning on a small set overfits and degrades held-out performance.

The threshold scales with model size: a 70B may need 50,000-plus, since it can absorb high-rank adapters before hitting the capacity ceiling.

Continued pre-training is the exception, measured in tokens rather than examples — beyond a few hundred million tokens of domain text, full fine-tuning is right regardless of model size.

The configurator walks the full setup:

Pick Full Fine-Tune as the adapter type — the rank, alpha, and target-module settings drop away, replaced by the core hyperparameters of full-parameter training.

Learning rates sit lower than LoRA’s: 1e-5 to 2e-5 for instruction fine-tuning, 1e-6 to 5e-6 for domain adaptation where the behaviour should shift without the base capabilities going with it. Use a cosine schedule with warmup to blunt the first updates.

Enable gradient checkpointing — activations are a big slice of the full-FT memory bill, and checkpointing typically removes 60 to 70 percent of them for 20 to 30 percent slower training. On long sequences or large batches it can be the difference between fitting and not.

Multi-device distribution is automatic: data parallelism, and tensor parallelism for the largest models — you set parameters, the platform picks the strategy.

Four moves, in order. Validate the hypothesis with a LoRA run on the same data first — if LoRA nears the quality target, the full run may not be worth its premium; if LoRA falls short, the delta justifies the spend.

Scale up through the family: prove the dataset and approach on a 7B or 13B before committing a 70B. Iterating on data quality, learning rate, and epochs is dramatically cheaper on the small model, and most lessons transfer.

Maximise batch size within the memory budget — checkpointing plus accumulation — since larger effective batches converge in fewer epochs and fewer epochs is a smaller bill.

And watch the validation loss with early stopping: once it stops improving, the run is done. Stop at the best checkpoint, not the scheduled end — every minute shaved is money kept.

The detail view streams the loss live, and a run can be stopped at any point:

Validate on adapters first — a rank-64 LoRA that meets the bar makes a full run unnecessary; one that falls short makes it justified. That gate is the cheapest quality insurance the method has.

Data quality matters more here than anywhere: with every parameter free to move, the model can memorise the noise. Deduplicate, cut low-quality examples, enforce formatting and style consistency — a clean 20,000 beats a noisy 100,000.

Cosine schedule with warmup on every run: the warmup (typically 3 to 10 percent of steps) prevents destabilising early updates that overwrite pre-trained knowledge, and the decay lets the model settle. Watch the first 10 percent of the loss curve closely — if it is not descending smoothly, stop and fix the learning rate before hours burn on a run that will not converge.

Checkpoint every 10 to 20 percent of steps. Runs are long and expensive, and when late-stage overfitting arrives you want the earlier checkpoint where validation loss was lowest, not a lost run.

When the dataset is small. Training every parameter on a few thousand examples is the classic road to catastrophic forgetting: the model learns your examples and unlearns part of what made it good. Below a meaningful size, LoRA at a sensible rank is not the cheap compromise — it is the better method.

When the gap is style, format, or voice. Surface behaviours are what adapters learn best; rewriting every weight in the model to change JSON formatting is the most expensive way to solve the cheapest problem in fine-tuning.

And when the result cannot be evaluated. A full fine-tune with no frozen test set to compare against the base model is a liability wearing a loss curve. No harness yet? Build it on adapters first — it will serve the full run when the full run is justified.

Start with data worth the method: at least 10,000 high-quality examples, verified in the preview before anything launches. New Training Run, Full Fine-Tune as the adapter type.

Learning rate 1e-5 to open, gradient checkpointing on, 2 to 3 epochs for the first run. Hardware selection and multi-device distribution happen automatically — nothing to configure.

The dashboard streams training loss, validation loss, learning rate, and throughput live. A smooth descent in both losses: let it complete and evaluate the checkpoint. Overfitting or divergence: stop early and adjust.

Per-second billing matters most on this method, where runs last hours: a run stopped at 45 minutes because the loss plateaued costs exactly 45 minutes of compute — no rounding up — which is what makes iterating on full fine-tunes feasible instead of betting everything on one long run.