.GGUF

GGUF model file

Introduction

GGUF is the model format of the ggml machine learning library and of llama.cpp, which runs large language models on ordinary computers. It was introduced in August 2023 to replace the earlier GGML files. One GGUF file holds everything needed to run a model: its weights, often quantised to 8, 5 or 4 bits per value to save memory, and metadata such as the architecture, the context length and the tokenizer’s vocabulary.

Software

Structure

Numbers are little-endian. A string is a 64-bit length followed by that many bytes of UTF-8.

  1. 0Magic the four bytes GGUF4 bytes
  2. 4Version currently 34 bytes
  3. 8Tensor count8 bytes
  4. 16Metadata count8 bytes
  5. 24Metadata key, value type, value; keys such as general.architecturerepeated
  6. …Tensor info name, number of dimensions, dimensions, type, offsetrepeated
  7. …Padding to the alignment, 32 bytes unless general.alignment says otherwisevaries
  8. …Tensor data each tensor at its offset from heremost of the file
A GGUF file: header, metadata, tensor list, then tensor data.

The type of each tensor names its encoding: plain floats such as F32 and F16, or one of the quantised block formats such as Q8_0 and Q4_K, which store small groups of values with a shared scale.