.JPG File Format – How the coding works

1. Colour

Colour images are usually stored as YCbCr rather than RGB, since the eye is less sensitive to detail in colour than in brightness. JPEG itself does not define colour: it treats components as plain numbers. The transform comes from JFIF or an Adobe segment, and JFIF gives it as:

Y  =  0.299 R + 0.587 G + 0.114 B
Cb = −0.1687 R − 0.3313 G + 0.5 B + 128
Cr =  0.5 R − 0.4187 G − 0.0813 B + 128

R = Y + 1.402 (Cr − 128)
G = Y − 0.34414 (Cb − 128) − 0.71414 (Cr − 128)
B = Y + 1.772 (Cb − 128)

Cb and Cr are usually kept at half or a quarter of the resolution of Y, which throws away colour detail straight away. The frame header gives the sampling factors.

2. Blocks and the level shift

Each component is cut into blocks of 8 × 8 samples. Edges are padded so whole blocks fill them: the padding is the encoder's own choice. Samples are level shifted by subtracting 2^P−1 (128 for 8-bit samples), so they run from −128 to 127.

3. The discrete cosine transform

Each block is replaced by 64 numbers that say how much of each of 64 fixed patterns it contains. Coefficient (0, 0) is the block's average, called DC. The other 63, called AC, hold finer and finer detail going right and down: the column gives the horizontal frequency and the row the vertical frequency.

S(v, u) = ¼ C(u) C(v) Σ Σ s(y, x) cos((2x + 1)uπ / 16) cos((2y + 1)vπ / 16)

s(y, x) = ¼ Σ Σ C(u) C(v) S(v, u) cos((2x + 1)uπ / 16) cos((2y + 1)vπ / 16)

C(0) = 1/√2, and C(u) = 1 otherwise

The two sums run over x and y (in the forward transform) or u and v (in the inverse), from 0 to 7. Written like this, a transform of 8-bit samples has DC values between −1,024 and 1,016 before quantisation.

4. Quantisation

Each coefficient is divided by the matching value of a quantisation table and rounded. This is where information is lost: the decoder can only multiply back. Many coefficients become zero, especially the high frequencies. The non-zero values, and how many of them there are, are what quality settings change.

5. Zig-zag order

The 63 AC coefficients are put in zig-zag order, which the quantisation page shows. High frequencies are mostly zero, so this order gathers the zeros into long runs at the end.

6. Coding the DC coefficient

DC values change little between neighbours, so the data carries the difference from the previous DC of the same component. The first difference of a scan, and of a restart interval, is from 0. Each difference is written as a Huffman code, for its size in bits (SSSS), then that many bits holding the value.

7. Coding the AC coefficients

The AC coefficients are written in zig-zag order as (run, value) pairs: a Huffman code for RRRR (the number of zeros before the value, 0 to 15) and SSSS (the value's size), then SSSS bits. A run of more than 15 zeros uses ZRL (F0), which stands for 16 zeros. The rest of a block, once it is all zeros, is one EOB code (00). A block whose last coefficient (63) is non-zero has no EOB.

The table to use comes from the scan header, and the tables' codes from the Huffman tables.

The bits of a value

A value of SSSS bits is written with its top bit showing its sign. Positive values are written as they are. A negative value v is written as v + 2SSSS − 1, which is its ones' complement. A decoder takes the SSSS bits as a number b:

if b < 2^(SSSS − 1):  value = b − 2^SSSS + 1
else:                 value = b

An example block

A block in which DC is 3 more than the block before it, with one AC coefficient of 1 in zig-zag place 2, is coded as:

The bits of those symbols come from the tables in the file. The coefficient at place 1 is the zero that the run skipped.