.HDS File Format – Header, BAT and extension

The header

64 bytes at the start of the file. QEMU’s and OpenVZ’s names for the fields differ; both are given.

QEMU’s probe accepts a file when the first 16 bytes are one of the two signatures and the version is 2. Disk size, cluster size and BAT size are tied together: the BAT must cover the disk, that is nb_bat_entries × tracks ≥ nb_sectors (QEMU: “Catalog size too small for advertised disk size”). When ploop makes a header it rounds the disk size up to a whole number of clusters, so nb_bat_entries = nb_sectors ÷ tracks.

The two signatures

Both are one format with the same header. ploop’s ploop1_version() tells the two apart by the signature alone (PLOOP_FMT_V1, PLOOP_FMT_V2); QEMU creates new images with the second.

The in-use word

Flags

The block allocation table (BAT)

The BAT starts at byte 64 and is a contiguous array of nb_bat_entries 32-bit unsigned entries, 4 bytes each. Entry n belongs to cluster n of the virtual disk, which holds the virtual bytes n × cluster size up to the next cluster.

• An entry of 0 means the cluster is not allocated and reads as zeros.
• Otherwise the entry says where the cluster is: its sector number in the file is the entry for WithoutFreeSpace, and the entry × tracks for WithouFreSpacExt. Its byte offset is that sector number × 512.

To read virtual byte b: c = b ÷ cluster size (rounded down), position p = b − c × cluster size, then read from the file at the byte offset of entry c, plus p.

Rules for the entries

QEMU’s page lists them; its open code and checker apply the first two (a cluster must lie inside the file and may not overlap the header or BAT):

• the value must not be lower than the data offset;
• the cluster must lie inside the file (QEMU: the sector number plus the cluster size must not pass the end of the file);
• the value must be unique among all BAT entries;
• the cluster’s offset minus the data offset must be a multiple of the cluster size.

Implementation limits of QEMU, not of the format: a cluster of more than 4,186,111 sectors (INT32_MAX ÷ 513) and a BAT of more than INT_MAX ÷ 4 entries are refused.

The data area and the padding

The data area runs from the data offset to the end of the file, as an array of clusters. Most are allocated by the BAT; some may be allocated by the header’s ext_off and by extensions. The bytes between the end of the BAT and the data area are padding.

• WithoutFreeSpace: if data_off is 0 the data area starts at the end of the BAT (64 + 4 × entries bytes) rounded up to a whole sector. If it is not 0 it should be aligned to the sector size, and it is recommended to align it to the cluster size for new images.
• WithouFreSpacExt: data_off must be non-zero and aligned to the cluster size.
• Either way data_off must be at least the end of the BAT in sectors (rounded up to a cluster for the second signature) and not past the end of the file; QEMU checks this and repairs it.

An image’s size must be a multiple of the cluster size. If it is not, ploop assumes the last extension of the file failed in the middle of a transaction, and starts new allocations at the size rounded down to a cluster. Updates to the BAT are made only after the data clusters are safely written, so a crash can leave an unused (lost) cluster but never a BAT entry that points at unwritten data.

The format extension

A cluster that carries optional features, found through the header’s ext_off. It must lie inside the file and meet the same requirements as a BAT cluster. QEMU drops (ignores) an extension that lies outside the file or whose magic or MD5 is wrong, and refuses to open the image if the extension is damaged inside while the image was closed correctly.

A feature section

The last section must be End of features, with every field 0 (ploop’s structure names the three parts BlockCheck, BlockElementHeader and, for the bitmap, DirtyBitmapExtension::Raw). A section’s data, rounded up to 8, must fit in what remains of the cluster, and there must be room for a 24-byte section header.

The dirty bitmap feature

Stores a QEMU-style dirty bitmap in the image: one bit for each granularity sectors of the disk, set when that part changed (used for incremental backup).

When an image was left open (in-use word Ynot), QEMU marks the stored bitmaps inconsistent.

What a valid image obeys

1. The signature is WithoutFreeSpace or WithouFreSpacExt, and the version is 2.
2. tracks is not 0; the BAT covers the disk (nb_bat_entries × tracks ≥ nb_sectors).
3. For WithoutFreeSpace the high 4 bytes of the size are 0.
4. The in-use word is 0x312E3276 or 0 for a closed image; 0x746F6E59 says it needs checking.
5. The data offset follows the rules for its signature, and is not before the end of the BAT or past the end of the file.
6. Every non-zero BAT entry is inside the file, at or after the data area, unique, and cluster-aligned from the data offset.
7. The format extension, if there is one, has its magic, a matching MD5, a final End of features, and features that fit inside the cluster.
8. Dirty bitmap: size equal to the disk size, a power-of-two granularity, an L1 table that fits, and L1 entries that point into the file.

Example: the start of a real image

The first 0x60 bytes of the expanding image of an Ubuntu 24.04 ARM64 machine made by Parallels Desktop (read directly from the file; the image is a 64 GiB disk in a 56.6 GB file):

00000000: 5769 7468 6f75 7446 7265 6553 7061 6365  WithoutFreeSpace
00000010: 0200 0000 1000 0000 0000 0400 0008 0000  ................
00000020: 0000 0100 0000 0008 0000 0000 7064 3137  ............pd17
00000030: 0008 0000 0000 0000 0000 0000 0000 0000  ................
00000040: 0010 0000 0018 0000 0020 0000 0010 ae00  ......... ......
00000050: 00b8 9b00 00b0 9b00 00a8 9b00 00a0 9b00  ................

Heads × cylinders × 32 sectors per track is the disk size in sectors, and 32 is the Sectors value of the image’s DiskDescriptor.xml. The header does not hold the sectors per track.