.ZST

Zstandard compressed file

Introduction

Zstandard, or zstd, is a compression method made by Yann Collet at Facebook and released in 2016 with the zstd tool. It compresses about as well as gzip at its fast settings and nearly as well as xz at its slow ones, while decompressing much faster than either. It is now common for software packages, such as Arch Linux’s .pkg.tar.zst, for file system and database compression, and for web traffic. The format is RFC 8878.

Zstandard can also use a dictionary trained on sample data, which helps a great deal when compressing many small, similar files or messages.

Structure

A .zst file is one or more frames one after another. Numbers are little-endian.

  1. 0Magic 28 B5 2F FD4 bytes
  2. 4Frame header a descriptor byte, then optional window size, dictionary ID and content size2 to 14 bytes
  3. …Blocks each a 3-byte header (last-block flag, type, size) and its datarepeated
  4. …Checksum low 32 bits of the XXH64 hash of the content, if the descriptor asks for it0 or 4 bytes
One Zstandard frame.

A block is raw (stored as it is), RLE (one byte repeated) or compressed. A compressed block holds literals coded with Huffman coding and sequences of matches coded with finite state entropy. A skippable frame, with a magic from 50 2A 4D 18 to 5F 2A 4D 18, carries other data that decompressors ignore.

Links

Software

  • zstd (macOS, Linux, Windows)
  • GNU tar, with the --zstd option (Linux, macOS)
  • bsdtar, from libarchive, the tar on macOS and Windows

Developers

Reference

  • RFC 8878, Zstandard Compression and the application/zstd Media Type