Adapter Types Compared: LoRA, QLoRA, Full Fine-Tune and Beyond

Adapters are small trainable modules inserted into a frozen pretrained model, letting fine-tuning run at a fraction of full-parameter memory and compute. The family has grown quickly, each method trading efficiency, expressiveness, and compatibility differently.

The platform supports multiple adapter types so the method can be matched to the constraints. The choice depends on available VRAM, dataset size, required quality, and inference deployment requirements.

LoRA injects low-rank matrices (A times B) into transformer layers — typically the attention projections (q, k, v, o) and MLP layers. At rank 16, a 7B model adds about 10-20M trainable parameters. Memory: approximately 16GB for 7B. Quality: excellent for most tasks. Speed: fast training, zero inference overhead after merging.

LoRA is the default recommendation for most fine-tuning scenarios: it balances quality, speed, and memory well, and produces adapters mergeable into the base weights for deployment without architectural changes.

QLoRA quantises the frozen base model to 4-bit NormalFloat precision and trains LoRA adapters in full precision on top. Memory: approximately 6-8GB for 7B. Quality: nearly identical to LoRA. Speed: 10-20% slower due to dequantisation in the forward pass.

Choose QLoRA when VRAM is the primary constraint. It enables fine-tuning 70B models on a single 48GB GPU — which would need 4+ GPUs with full-precision LoRA.

Full fine-tuning updates every parameter. Memory: approximately 56GB for 7B (four times the model size, for weights plus optimiser plus gradients). Quality: the highest ceiling with sufficient data. Speed: comparable to LoRA per step, often needing fewer total steps.

Reserve it for cases where adapter methods underperform: domain shifts requiring deep knowledge changes, very large datasets (100K+ examples), or work needing changes to the model’s fundamental capabilities.

AdaLoRA allocates the rank budget across layers dynamically, by importance scores during training: layers benefiting from higher rank get more capacity, less important layers use lower rank. With the same total parameter budget, this can beat fixed-rank LoRA.

AdaLoRA adds slight overhead for the importance-scoring mechanism, but can improve results on tasks where layers contribute unevenly to output quality.

LoHa (Low-Rank Hadamard) uses a Hadamard product of two low-rank matrices instead of a matrix product — a different parameterisation that can be more expressive per parameter. BOFT (Butterfly Orthogonal Fine-Tuning) constrains weight updates to orthogonal transformations via butterfly matrices, preserving pretrained weight norms and preventing catastrophic forgetting. ReFT (Representation Fine-Tuning) operates on hidden representations rather than weights, intervening in the forward pass to steer activations.

Newer methods, narrower use cases. LoHa originated in image generation and helps when LoRA at a given rank is insufficient; BOFT shows promise for continual learning; ReFT is particularly parameter-efficient but demands careful layer selection.

Open with LoRA (rank 16) as the default. VRAM tight? Switch to QLoRA. Quality insufficient at LoRA rank 64? Consider full fine-tuning with sufficient data and compute. AdaLoRA for automatic rank allocation without manual experimentation.

The wizard carries the adapter selection and the method-specific parameters:

Comparing adapters is quick on the platform: short training jobs with different adapter types on the same dataset, loss curves and evaluation metrics compared directly.

The training list holds every adapter comparison experiment:

Isolating the adapter’s effect from every other variable demands controlled experiments: only the adapter type changes, while dataset, base model, learning-rate schedule, batch size, and epochs stay constant. Straightforward on the platform — multiple runs with identical configurations except the adapter selection.

Start with three baselines: LoRA rank 16, QLoRA rank 16, LoRA rank 64 — covering standard efficiency, maximum memory savings, and high-capacity adaptation. Compare final validation loss and task-specific metrics across all three. LoRA rank 16 meeting the bar means no further exploration; falling short means trying rank 64 before moving to full fine-tuning.

Look beyond the final loss value: the shape of the curves matters — smooth and steadily decreasing indicates stable training, while oscillations or plateaus suggest suboptimal hyperparameters. And compare inference quality on a held-out test set with task metrics: two runs with similar final loss can differ noticeably on real examples.

Per-second billing keeps adapter comparison cheap: each short run (100 to 500 steps) costs minutes of compute. Five quick comparison configurations cost far less than committing to one and discovering it suboptimal after a full run.

When the change must reach the model’s core knowledge. Adapters steer; they do not re-educate. A model that has never seen the domain’s corpus needs continued pre-training or retrieval, not a cleverer low-rank update.

When a benchmark is being chased rather than a behaviour. Adapter methods trade a little expressiveness for large savings in memory and cost — and on evaluations stressing the limits of capability, that trade shows. A target genuinely at the capability frontier deserves full fine-tuning as the honest comparison.

And when the serving plan cannot carry them. Adapters are cheap to train and cheap to merge — but a deployment story built on swapping unmerged adapters at request time adds an operational layer. Merge before serving, serve the merged artifact, and keep the runtime simple.

All the major adapter types live in one place, so switching methods on the same dataset is a small configuration change rather than a new pipeline. Upload the dataset, open Training, create a new run, and pick the base model from the catalog.

The adapter step presents the choice: LoRA and QLoRA carry rank, alpha, and target-module settings; AdaLoRA adds its rank-budget parameters; the exotic methods surface their own few options; full fine-tuning drops the adapter settings entirely in favour of core hyperparameters. Defaults are sensible throughout, and the memory estimate updates with the selection before launch.

After the run, the same training list and detail views compare adapters side by side — loss curves, validation metrics, checkpoints — so the honest comparison above is a workflow, not a project.