A sparse file has holes: long runs of zeros that the file system does not store. GNU
tar (with --sparse) writes such a file as a member of type S
that stores only the parts holding data. A sparse map lists each part as an offset in the
original file and a length; the member’s data is those parts one after another.
0Header typeS, up to four map entries,isextended,realsize512 bytes512Extension blocks only ifisextendedis set: 21 more map entries each512 bytes each…Data the stored parts in map order,sizebytes, padded to a multiple of 512size rounded up
The sparse header
The header is an ordinary GNU
header of type S (GNUTYPE_SPARSE). Its size
is the number of bytes stored, not the size of the file. The sparse fields are:
| Offset | Size | Type | Name | Description |
|---|---|---|---|---|
386 | 12 | octal | sparse[0].offset | Where the first stored part starts in the original file. |
398 | 12 | octal | sparse[0].numbytes | How many bytes it holds. |
410 | 24 | 2 × octal | sparse[1] | The second part: offset, then length. |
434 | 24 | 2 × octal | sparse[2] | The third part. |
458 | 24 | 2 × octal | sparse[3] | The fourth part. |
482 | 1 | flag | isextended | Non-zero when extension blocks with more entries follow the header. |
483 | 12 | octal or base-256 | realsize | The file’s full size, holes included. |
Entries are used in order. An entry that is not used is left as zeros; a map that
needs fewer than four entries leaves the rest empty and isextended zero.
Extension blocks
When four entries are not enough, isextended is set and extension blocks
follow the header, before the data. Each is a 512-byte block in this layout
(sparse_header in tar.h):
| Offset | Size | Type | Name | Description |
|---|---|---|---|---|
0 | 504 | 21 × 24 bytes | sparse | 21 more map entries, each a 12-byte octal offset and a 12-byte octal length. |
504 | 1 | flag | isextended | Non-zero when another extension block follows this one. |
505 | 7 | padding | – | Zeros. |
To read a sparse member:
- Read the up to four entries in the header.
- While the last
isextendedread is non-zero, read the next block as an extension block and add its entries. Go’s reader stops whenisextendedis zero, noting that “this termination condition is identical to GNU and BSD tar”. - The data starts at the next block. Its length is
size, which is the sum of the entries’ lengths. - To rebuild the file, make
realsizebytes of zeros and copy each stored part, in turn, to its offset.
A reader that does not know S extracts the stored parts joined together
as one shorter file. In pax archives GNU tar writes sparse files differently, with
GNU.sparse.
records; star’s sparse header has
yet another layout under the
same type flag.
Example
The member sparse-gnu, the first in Go’s test archive
sparse-formats.tar. Its header starts at 0x000:
| At | Field | Bytes | Value |
|---|---|---|---|
0x182 | sparse[0].offset | 30 30 30 30 30 30 30 30 30 30 31 00 | 1 |
0x18E | sparse[0].numbytes | 30 30 30 30 30 30 30 30 30 30 31 00 | 1 |
0x19A | sparse[1] | – | offset 3, 1 byte |
0x1B2 | sparse[2] | – | offset 5, 1 byte |
0x1CA | sparse[3] | – | offset 7, 1 byte |
0x1E2 | isextended | 01 | extension blocks follow |
0x1E3 | realsize | 30 30 30 30 30 30 30 30 33 31 30 00 | octal 310, 200 bytes |
Five extension blocks follow, from 0x200 to 0xBFF, and then
95 bytes of data: the file is 200 bytes long, and only its stored parts are in the
archive.