LoRA (Low-Rank Adaptation)

Definition. A parameter-efficient fine-tuning technique that freezes the base model and injects small trainable low-rank matrices into selected layers, typically the attention projections. Instead of updating billions of parameters it updates a few million, which reduces GPU memory by an order of magnitude and produces an adapter file of tens of megabytes that can be loaded and unloaded at runtime.

How it works. Pick target modules (query, key, value and output projections are standard), set a rank typically between 8 and 64, and an alpha scaling factor usually twice the rank. Train with a modest learning rate and a small number of epochs, watching validation loss for overfitting. Save the adapter, not the full model. At serving time load the base model once and swap adapters per tenant or per task.

Example. A platform serves twelve enterprise customers, each wanting a different tone and output schema. Rather than running twelve full models, it runs one base model and twelve LoRA adapters of 40 MB each. Switching adapters takes milliseconds, GPU memory holds the base plus one adapter at a time, and adding a thirteenth customer costs a two-hour training run instead of a new deployment.

Use case. LoRA is the standard approach for multi-tenant customisation, rapid experimentation, and fine-tuning on a single consumer GPU. It is also used to adapt open models to a domain such as clinical notes or legal drafting without the cost of full retraining.

Warnings. Adapters are tied to a specific base model version; upgrading the base invalidates them. Merging an adapter into the base weights is irreversible and prevents runtime swapping, so keep the unmerged adapter as the source of truth.

Limitations. LoRA has less capacity than full fine-tuning and can underfit on tasks that require substantial new behaviour. Rank and target-module choices are empirical, and quality varies unpredictably across tasks. It also does not solve the underlying knowledge-staleness problem.

Pros and cons
ProsCons
Supports lighter-weight fine-tuning and multiple adapters.The adapter can underfit tasks requiring substantial new behavior.
← Browse all glossary terms