.GGUF

Introduction

GGUF is the model format of the ggml machine learning library and of llama.cpp, which runs large language models on ordinary computers. It was introduced in August 2023 to replace the earlier GGML files. One GGUF file holds everything needed to run a model: its weights, often quantised to 8, 5 or 4 bits per value to save memory, and metadata such as the architecture, the context length and the tokenizer’s vocabulary.

A GGUF file can also hold a LoRA adapter instead of a whole model: a small set of extra weights that adjust a base model. llama.cpp converts adapters from the Hugging Face LoRA layout to GGUF and applies them to the base model when it loads.

Structure

Numbers are little-endian. A string is a 64-bit length followed by that many bytes of UTF-8.

The type of each tensor names its encoding: plain floats such as F32 and F16, or one of the quantised block formats such as Q8_0 and Q4_K, which store small groups of values with a shared scale.

Version 1 files used 32-bit counts and string lengths; version 2 widened them to the 64 bits shown above, and version 3 added support for big-endian files. Current software writes version 3.

Links