.PDF File Format – Streams and filters

The stream dictionary

Every stream dictionary can hold these keys, whatever the stream is for:

Streams of particular kinds add their own keys, such as Width for an image or Length1 for an embedded font. With two filters,

<< /Length 534 /Filter [/ASCII85Decode /LZWDecode] >>

Length is what the file holds, not the decoded size. A reader that finds endstream somewhere other than where Length says has a damaged file; many readers then trust the keyword and use the bytes up to it.

The filters

The short names are used only in inline images inside content streams. The image filters produce pixels for an image and are not used on other streams. The order in Filter is the order of decoding: the first filter listed was the last applied when writing.

ASCIIHexDecode

Each byte is written as two hexadecimal digits, upper or lower case. White space is ignored. > ends the data; if the number of digits before it is odd, the last digit is taken as followed by 0. The data doubles in size. So 48 65 6C 6C 6F> decodes to Hello.

ASCII85Decode

Each group of four bytes is read as a 32-bit number, high byte first, and written as five digits in base 85, each digit plus 33, which gives the characters ! to u. The data grows by a quarter.

• A group of four zero bytes is written z instead of !!!!!.
• At the end, a last group of one to three bytes is padded with zeros to four, encoded, and only the first n + 1 characters written for n bytes. The decoder pads the characters with u and keeps n bytes.
• ~> ends the data. White space is ignored; any other character is an error.

The four bytes Man (4D 61 6E 20) are the number 1,298,230,816, which in base 85 is the digits 24, 73, 80, 78, 61: the characters 9jqo^.

LZWDecode

The adaptive Lempel–Ziv–Welch method used by TIFF. Codes are 9 to 12 bits long, high bit first. A table of strings starts with the 256 single bytes; code 256 clears the table and returns to 9-bit codes, and 257 ends the data. Each new code adds the previous string plus the first byte of the next to the table, from code 258.

Before PDF 1.2 LZW was the only general compression, so files from Acrobat 1 and 2 use it for content streams. Flate, added in PDF 1.2, usually compresses better and was not held back by the LZW patents of the time.

FlateDecode

From PDF 1.2: the data is a zlib stream (RFC 1950) holding deflate data (RFC 1951). It starts with a two-byte header, usually 78 9C or 78 DA, and ends with an Adler-32 checksum. It is the usual compression for content streams, fonts and images that are not JPEG. Some writers leave off the checksum or add bytes after it; readers usually accept what they could inflate.

RunLengthDecode

The data is a series of runs, each starting with a length byte:

02 41 42 43 FE 2A 80 decodes to ABC***: three bytes copied, then 2A repeated 257 − 254 = 3 times.

CCITTFaxDecode

One-bit images compressed as fax machines do, by ITU-T recommendations T.4 (Group 3) and T.6 (Group 4). The data has no header, so the parameters say how it was written:

The output is rows of packed bits, each row padded to a whole byte.

JBIG2Decode

From PDF 1.4: one-bit images compressed by JBIG2 (ISO/IEC 11544 and ITU-T T.88), which can store the shapes of text characters once and place them many times. The stream holds the segments of one page in the embedded form of the standard, with no file header. Segments shared by several images, such as a symbol dictionary, can be in a separate stream:

1 bits are black, unlike CCITT’s default. The page segment’s size should match the image’s Width and Height.

DCTDecode

JPEG data, as in a JPEG file, from the FF D8 start-of-image marker to FF D9. Baseline and progressive JPEG with 1, 3 or 4 components are allowed. The image dictionary’s Width, Height, ColorSpace and BitsPerComponent (8) must match the JPEG.

CMYK JPEGs written by Adobe programs often store the components inverted, which a reader sees from the APP14 marker or from a Decode array of [1 0 1 0 1 0 1 0] on the image.

Predictors

LZW and Flate compress images better if each sample is first replaced by its difference from a neighbour. The Predictor parameter says which method was used, and the decoder undoes it after decompressing:

The PNG predictors came with Flate in PDF 1.2. For any value from 10 to 15 the data is the same: every row starts with a byte that names the PNG filter used on that row, 0 None, 1 Sub, 2 Up, 3 Average, 4 Paeth, so the decoder follows that byte whatever the value. A row has Colors × BitsPerComponent × Columns bits, rounded up to whole bytes, plus the filter byte. Each predicted byte is added, modulo 256, to:

• Sub: the byte one sample to the left (one byte for 8-bit grey, three for 8-bit RGB), or 0 at the start of the row;
• Up: the byte above it in the previous row, 0 on the first row;
• Average: half the sum of the left and above bytes, rounded down;
• Paeth: whichever of left, above and upper left is nearest to left + above − upper left.

An example with Up and Columns 3, after inflating: the rows 02 01 02 03 and 02 00 01 FF decode to 01 02 03 and 01 03 02, because the second row adds each byte to the one above it, and 03 + FF is 02 modulo 256.

External streams

From PDF 1.2 a stream’s data can live in another file, named by F. The data in the PDF itself is then ignored and is usually empty, and FFilter and FDecodeParms describe the external file. It was meant for large images and sounds kept beside a document. Few readers support it, and a file sent on its own loses the data.