.SAFETENSORS

Introduction

LoRA, short for low-rank adaptation, is a way to fine-tune a large model by training a small set of extra weights instead of changing the model itself. It was described by researchers at Microsoft in 2021. A LoRA adapter file holds only those extra weights, so it is a small fraction of the size of the base model it was trained for, and it does nothing on its own. Image generators and language models both use them, to add a style, a character, a subject or a task.

Adapters are almost always shared as safetensors files. What makes one a LoRA is the names and shapes of the tensors inside. Older ones were also saved as PyTorch .pt or .bin files, and llama.cpp converts them to GGUF.

Structure

For each layer it adapts, such as an attention projection with a weight matrix W of d outputs by k inputs, the adapter stores two small matrices: A, of r × k, and B, of d × r. The rank r is small, often between 4 and 128. When the adapter is applied, the layer’s weight becomes W + (alpha / r) · BA. alpha is a scaling number chosen at training time, and programs usually let the user scale the effect further.

Two naming schemes are common. Adapters made with Hugging Face PEFT come as adapter_model.safetensors with keys such as base_model.model.model.layers.0.self_attn.q_proj.lora_A.weight and ….lora_B.weight, and a separate adapter_config.json giving r, lora_alpha, target_modules and the base model’s name.

Stable Diffusion LoRAs made with the kohya-ss scripts keep everything in one file. Their keys start lora_unet_ for the image model or lora_te_ for the text encoder, followed by the layer path with underscores, and end .lora_down.weight (A), .lora_up.weight (B) and .alpha, a one-number tensor per layer. Training settings are often kept as text in the safetensors __metadata__.

Also used for

• Safetensors .safetensors

Links