.TAR File Format – Pax extensions

Extended headers

An extended header is an ordinary 512-byte header with type flag x, followed by data blocks holding the records. Its own size is the length of the records. It applies to the member straight after it: POSIX’s order is the optional x header and its data, then the ustar header the records apply to, then that member’s data. Its records replace the matching fields of the member’s header.

The extended header’s own name is not used for anything. POSIX’s default is %d/PaxHeaders.%p/%f: the member’s folder, PaxHeaders. and the process ID, then its file name. So a program that does not understand pax extracts it as a small, recognisable file. GNU tar writes names in this form; Go writes PaxHeaders.0/ and the name.

Solaris tar wrote extended headers with type X. Readers such as libarchive, and the explorer, treat them as x.

A worked example, from Go’s test archive pax.tar: the x header is named a/PaxHeaders.6887/123456789101112…, its size is 00000000446 (294 bytes) and its type flag at 0x9C is 78. Its data starts at the next block:

00000200  32 30 34 20 70 61 74 68 3D 61 2F 31 32 33 34 35  204 path=a/12345

Global headers

A header with type flag g holds records in the same form, but they apply to every member that follows, until another g header changes them. A g header can appear anywhere in the archive, and there may be several. Its records give defaults: a record in a member’s own x header takes priority over a global one.

POSIX’s default name for a global header is $TMPDIR/GlobalHead.%p.%n: the temporary folder, the process ID and a sequence number. Go writes GlobalHead.0.0.

Records

Each record is a line of text, in the form "%d %s=%s\n":

30 mtime=1730851200.123456789
51 path=a/very/long/directory/name/report-2024.pdf

The line is a decimal length, a space, the keyword, =, the value and a line feed. The length counts the whole record in bytes, including the digits of the length itself, the space and the line feed. In the first record above, mtime=1730851200.123456789 is 26 bytes; with the two digits of 30, the space and the line feed, the whole record is 30 bytes. Because of the length, a value can contain any byte, line feeds included.

To read the records:

1. Read decimal digits up to a space: the length n.
2. The record is the n bytes from the first digit. Its last byte must be a line feed.
3. After the space, the keyword runs to the first =; keywords are UTF-8 and cannot contain =.
4. The value is everything after the = up to the final line feed.
5. Move on n bytes and repeat, until the header’s size is used up.

Keywords are UTF-8. Values are UTF-8 unless a hdrcharset record says otherwise. Numbers are decimal text of any length, and times are seconds since 1970 with an optional fraction. A record with an empty value deletes the field, or the value an earlier record set.

Keys

The keywords POSIX defines:

ctime, the status change time, is not in POSIX’s list, but GNU tar and Go write it and libarchive reads it.

When the same value is set in several places, POSIX gives this order for reading and listing; the first that applies wins:

1. a -o delete= option of the pax command;
2. a -o keyword:= option;
3. a -o keyword:=value option;
4. a record in the member’s x header (the last one, if the keyword appears twice);
5. a -o keyword=value option;
6. a record in a g header (again the last one);
7. the ustar header itself.

Sparse files

GNU tar writes sparse files in pax archives with GNU.sparse. records. Three versions exist, described in libarchive’s tar(5) and read by Go. Which GNU tar release brought each one is not confirmed here.

In version 1.0 the member’s data starts with the map: decimal numbers, each ended by a line feed, first the number of entries, then the offset and size of each stretch of data, padded with zeros to a whole block. The stored stretches follow. Go notes that “the GNU manual says that numeric values should be encoded in octal format. However, the GNU tar utility itself outputs these values in decimal.” The member’s own header gets a name such as ./GNUSparseFile.1234/sparse-posix-1.0, and GNU.sparse.name holds the real one.

An example from Go’s test archive sparse-formats.tar. The extended header’s records start with the version:

00003400  32 32 20 47 4E 55 2E 73 70 61 72 73 65 2E 6D 61  22 GNU.sparse.ma
00003410  6A 6F 72 3D 31 0A 32 32 20 47 4E 55 2E 73 70 61  jor=1.22 GNU.spa

and the member’s data starts with the map:

00003800  39 35 0A 31 0A 31 0A 33 0A                       95.1.1.3.

GNU tar’s own format stores sparse files differently, in the header; see GNU sparse files.

Vendor keys

Programs add their own keywords under a prefix in capitals. A reader skips keywords it does not know.