LoRA

last edited

LoRA (low-rank adaptation) trains two small matrices beside each frozen layer of a model. Their product is added to that layer, so a LoRA file of a few hundred megabytes can change a checkpoint worth tens of gigabytes. The method comes from a 2021 paper and now adapts language, image and video models alike.

How it works

For each targeted layer, LoRA training learns a tall and a wide factor whose product has a low rank, often 16 to 128. Only the LoRA factors receive gradients, so memory and file size stay small. At load time the product is scaled and added, and several LoRAs can be mixed at separate strengths, as in multi-LoRA stacking.

Language models

LoRA began as a language-model method. The original LoRA paper cut trainable parameters by about 10,000 times. QLoRA (2023) trains adapters over a 4-bit quantized model, putting large finetunes on one GPU. Servers such as vLLM keep one shared base model loaded and pick a LoRA per request.

Image and video

Image LoRAs teach a style, character or concept, usually from a few dozen captioned images with tools like AI Toolkit and sd-scripts. Video LoRAs add motion, camera moves or whole tasks, such as the LTX-2.5 in-context LoRAs that turn layouts into frames. Step-distilled turbo versions of new models also often ship as LoRAs.

Sources