LoRA vs QLoRA: Parameter-Efficient Fine-Tuning Explained

LoRA — Low-Rank Adaptation — adapts a model without updating all of its parameters. It came out of the paper “LoRA: Low-Rank Adaptation of Large Language Models” and has become the default for production fine-tuning, because it lands close to full fine-tuning on quality while spending a fraction of the compute and memory.

The insight underneath it: the weight updates a fine-tune applies have low intrinsic rank — a product of two small matrices can stand in for a full-sized update. A matrix with millions of entries gets its effective update from two matrices of a few hundred entries each, with minimal quality loss.

In practice adapters run under one percent of the base model’s parameter count: cheap to store, fast to swap, easy to version. You keep several adapters — one per task or domain — and load whichever applies, without duplicating the base model.

Mechanically, take a weight matrix W with dimensions d by k, say, an attention projection matrix in a transformer. During normal training, updating W requires storing a gradient of the same size d by k, which is expensive for large models. LoRA instead introduces two smaller matrices: A with dimensions r by k and B with dimensions d by r, where r is the rank and is typically much smaller than both d and k (common values are 8, 16, 32, or 64).

During the LoRA forward pass, the effective weight is W plus the product BA, where BA has the same shape as W but is parameterized by far fewer values: d times r plus r times k instead of d times k. The matrix A is initialized randomly (using a normal distribution) and B is initialized to zero, so the initial contribution of the LoRA update is zero. Training starts from the pre-trained behavior and learns from there.

The rank r is the primary hyperparameter controlling LoRA capacity. Higher rank means more parameters and more representational capacity, approaching full fine-tuning at the limit. Lower rank means fewer parameters, faster training, and smaller adapters, but may not capture all the necessary fine-tuning signal for complex tasks. Alongside rank, the alpha parameter scales the LoRA update: the actual update applied is (alpha/r) times BA. A common convention is to set alpha equal to twice the rank. The target_modules parameter specifies which weight matrices receive LoRA adapters, typically the attention query, key, value, and output projection matrices, and sometimes the feed-forward layers as well.

QLoRA — Quantized LoRA — quantises the base model to cut memory further. Standard LoRA keeps the base in bfloat16 or float16; QLoRA drops it to 4-bit via NF4 (NormalFloat4) before the adapters go on.

NF4 rather than plain integer quantisation because it is built for normally distributed weights — which well-trained networks have — arranging 16 levels to minimise expected error across a normal distribution. Same bit budget, less information lost.

The adapters themselves train and store at higher precision, typically bfloat16. The asymmetry is the trick: the frozen base takes quantisation without quality loss because it is not being updated, while the small active adapters need the precision for stable gradients. Net effect: roughly 50 to 70 percent less memory than standard LoRA, at quality usually within a few percent — often indistinguishable.

For a 7B model: full fine-tuning in bfloat16 needs about 56 GB — weights, gradients, and optimiser states. LoRA at rank 16 brings it to roughly 24 GB, because gradients and optimiser states exist only for the small adapters. QLoRA lands at about 12 to 14 GB with the base in 4-bit.

For 13B the numbers roughly double — around 104 GB full, 48 GB LoRA, 24 GB QLoRA. For 70B: full fine-tuning around 560 GB (impractical on most single nodes), LoRA about 240 GB, QLoRA a workable 120 GB. Figures are approximate — sequence length, batch size, and checkpointing move them.

Gradient checkpointing trades compute for memory, recomputing activations in the backward pass instead of storing them: typically 30 to 40 percent less memory for 20 to 30 percent slower training, and it can be enabled independently of the adapter.

The wizard shows adapter options and memory estimates before launch:

When memory is not the binding constraint, standard LoRA is the better default: no quantisation overhead, marginally higher quality on average, and adapters that need no quantised base at inference.

QLoRA wins when memory is the constraint — the classic case being a model too large for standard LoRA to fit, a 70B on limited hardware. It also buys parallelism: the smaller footprint lets several experiments share one machine.

Quality differences are small and task-shaped. On most tasks QLoRA is indistinguishable from LoRA; on precise numeric work, long-range dependencies, or subtle stylistic control, LoRA can hold an edge. The working pattern: prototype with QLoRA, and where a gap shows on your task, A/B against standard LoRA to see if it matters.

The run configuration form carries the parameters that matter:

• Adapter type: LoRA (bfloat16 base) or QLoRA (4-bit NF4 base).
• Rank (r): 8 for tight budgets, 16 as the general baseline, 32 to 64 for complex tasks or large datasets.
• Alpha: conventionally twice the rank — r 16, alpha 32 — giving the update an effective scale of 2.0, a default that has held across many tasks.
• Target modules: at minimum the attention projections — q_proj, k_proj, v_proj, o_proj. Adding the feed-forward layers (gate_proj, up_proj, down_proj on LLaMA-family models) buys quality at the cost of more adapter parameters.
• LoRA dropout: 0.05 to 0.1 for regularisation.

Loss curves, the learning-rate schedule, and checkpoint history stream on the training detail page:

Open at rank 16, alpha 32 — the middle ground that works broadly. Loss curves healthy and metrics near target? Stay. Quality short? Try rank 32 or 64. Memory or time tight? Rank 8.

Always include the attention projections in target modules — most of the model’s contextual computation happens there, and it is where LoRA pays most. Feed-forward layers earn their place for knowledge injection or behavioural change; skip them for lighter style adaptation.

Learning rates: standard LoRA sits well at 1e-4 to 2e-4; QLoRA at 1e-4 to 3e-4 — quantisation shifts the gradient dynamics slightly and tolerates a touch more. Gradient checkpointing cuts memory for modest compute, especially valuable where QLoRA already runs near the limit.

Run a short smoke eval before committing hours: a 100-step launch with the full configuration, checking the loss descends and training stays stable. Catches configuration errors before they spend compute. When a checkpoint earns its place, merge the adapter into the base weights for a clean single-file model — inference then carries no adapter overhead.

When the gap is knowledge, not behaviour. Adapters reshape how a model responds; they are a poor vehicle for teaching it facts it never learned. “The model does not know” is a retrieval problem or a continued-pre-training problem — a rank-16 adapter will not fix it.

When the behaviour you need is one prompt away. If a clear system prompt and two examples hold the format and the voice, every adapter is a pipeline maintained for a difference users cannot feel. Prompt first; train when the prompt demonstrably is not enough.

And when the change must reach the model’s core capabilities — a genuinely new skill, a deep domain shift. Low-rank updates have a ceiling, and the honest tool above it is full fine-tuning. The memory ladder is for climbing past rungs, not standing on the first one forever.

Both run on any supported model, side by side. Upload the dataset in the Datasets section — validated immediately, previewable before training — then open New Training Run for the seven-step wizard.

The adapter step offers LoRA or QLoRA, and the wizard adjusts the hyperparameters to the choice. LoRA asks for rank, alpha, target modules, dropout; QLoRA carries the same plus quantisation handled automatically — the base loads in 4-bit NF4 with nothing further to configure.

The training list tracks active and finished runs with status and key metrics, so comparing several LoRA and QLoRA experiments costs no extra work — and because billing is per second, ranks 8, 16, and 32 can be tried in quick succession to find the configuration your task wants.

Loss flat after a few hundred steps: the learning rate is usually too low — raise it 2 to 5×. Loss oscillating or spiking: too high — halve it and restart.

Out-of-memory mid-run: on standard LoRA, switch to QLoRA to shrink the base; already on QLoRA, cut batch size, enable gradient checkpointing, or shorten the sequence. Lowering rank helps memory too, at a possible quality cost.

Fine-tuned model worse than the base: the usual suspects are overfitting (too many epochs on a small set), a learning rate high enough to overwrite pre-trained knowledge, or weak data. Read the validation curve — validation loss rising while training loss falls means cut epochs or add regularisation — and re-sample the data to confirm the outputs are the quality you want reproduced.

QLoRA specifically lagging LoRA on the same task: raise the rank. Quantisation trims a little of the capacity available for adaptation, and a higher rank compensates — 16 to 32 often closes the gap entirely.