.SWF File Format – Compression

The LZMA fields

The layout is as 7-Zip’s SWF handler, Ruffle, Mozilla bug 775743 and Suricata read it: 7-Zip’s handler sets CompressedLength to the bytes after the 17-byte header when it writes, and bounds its decoding by FileLength − 8. The explorer shows CompressedLength, then the properties and dictionary size as one group, then the compressed bytes the decoder used as one range of the file.

LZMA properties

The properties byte packs three numbers:

• lc = Properties mod 9: how many high bits of the previous byte select the literal coder (0 to 8)
• lp = (Properties ÷ 9) mod 5: how many low bits of the output position select it too (0 to 4)
• pb = Properties ÷ 45: how many low bits of the position select the position state for matches (0 to 4)

The usual value is 5D, 93 = 3 + 9 × (0 + 5 × 2): lc 3, lp 0, pb 2. A byte above 224 (9 × 5 × 5 − 1) is not valid; the explorer turns such a file down. The dictionary size that follows is a UI32; the reference decoder treats anything below 4,096 as 4,096.

The range decoder

LZMA codes everything as single bits, each with a probability that adapts as the data goes. The range decoder keeps two 32-bit numbers, range and code.

1. Start: the first byte of the data must be 0. The next four bytes, most significant first, make code; range is 0xFFFFFFFF.
2. Every probability starts at 1,024, a half in 11-bit fixed point (2,048 is 1).
3. To decode a bit with probability p: bound = (range >> 11) × p. If code < bound, the bit is 0: range = bound and p += (2,048 − p) >> 5. Otherwise the bit is 1: range −= bound, code −= bound and p −= p >> 5.
4. Normalise: whenever range falls below 2²⁴, shift range and code left by 8 bits and bring the next input byte into code.
5. Direct bits, with no probability (used for the middle bits of long distances): range >>= 1; if code ≥ range, the bit is 1 and code −= range; then normalise.

Numbers of several bits are decoded through bit trees: start with m = 1; for each of n bits, m = 2m + the bit decoded with probability number m; the value is m − 2ⁿ. So each bit has its own probability, depending on the bits before it. A reverse bit tree works the same way but puts the bits into the value lowest first.

The decoding loop

The decoder keeps a state from 0 to 11 that remembers what the last few packets were (below 7: the last was a literal), four repeated distances rep0 to rep3, all starting at 0, and the output so far, which is also the dictionary. The position state is the output position’s low pb bits. Until FileLength − 8 bytes are out:

1. Decode an isMatch bit, whose probability is chosen by the state and the position state.
2. 0: a literal. Decode one byte and append it. The state becomes 0 if it was below 4, state − 3 if it was below 10, otherwise state − 6.
3. 1: decode an isRep bit (chosen by the state).
   • 0, a match: shift the distances (rep3 = rep2, rep2 = rep1, rep1 = rep0), decode a length with the match length coder, then a new rep0 with the distance coder. The state becomes 7 if it was below 7, otherwise 10.
   • 1, a repeated match: decode isRepG0. If 0, decode isRep0Long (chosen by state and position state); if that is 0 too, this is a short rep: copy one byte from rep0, and the state becomes 9 (or 11 if it was 7 or more). Otherwise the distance is rep0, or, through isRepG1 and isRepG2, rep1, rep2 or rep3, which is moved to the front of the four. Decode a length with the repeated-match length coder; the state becomes 8 (or 11).
4. For a match or repeated match, copy length + 2 bytes, one at a time, from rep0 + 1 bytes back in the output. Copies may overlap what they write.

A match that reaches back before the start of the output, or beyond the dictionary size, is an error; the explorer stops there and says so.

Literals

Literal bytes are decoded with one of several sets of 0x300 probabilities. The set is chosen by the low lp bits of the output position and the high lc bits of the previous byte: number ((position & (2^lp − 1)) << lc) + (previous byte >> (8 − lc)). With lc 3 and lp 0, the top three bits of the previous byte pick one of 8 sets.

• After a literal (state below 7), the byte is an 8-bit bit tree in the set’s first 256 probabilities.
• Straight after a match (state 7 or more), the decoder uses the match byte, the byte at distance rep0, as a guess: bit by bit, from the top, it decodes with probabilities chosen by the match byte’s bit as well. As soon as a decoded bit differs from the match byte’s, it carries on with the plain tree for the remaining bits.

Lengths

There are two length coders, one for matches and one for repeated matches, built the same way. Each decodes a number from 0 to 271, which is length − 2:

So a match is 2 to 273 bytes long.

Distances

1. Pick the length state: the decoded length value, at most 3. Decode a 6-bit distance slot with that length state’s tree (64 slots).
2. Slots 0 to 3 are the distance itself.
3. From slot 4: the number of extra bits is (slot >> 1) − 1, and the distance starts as (2 + (slot & 1)) << extra bits.
4. Slots 4 to 13: add the extra bits, decoded with a reverse bit tree of their own.
5. Slots 14 to 63: add the extra bits but the lowest 4 as direct bits, shifted left by 4, then the lowest 4 bits with the shared 4-bit reverse “align” tree.
6. The result is rep0; the bytes are copied from rep0 + 1 back. A distance of 0xFFFFFFFF is the end marker.

The end of the data

LZMA data may end with an end marker, a match whose distance is 0xFFFFFFFF, after which the range decoder’s code should be 0. Whether ZWS writers include one is not settled. 7-Zip’s handler does not need it, since FileLength gives the size, and a comment in it notes that some files have 8 extra bytes at the end that are not part of the stream. The swfzip tool describes the data as ending with a 6-byte end marker that CompressedLength includes; Mozilla’s change for bug 775743 adds an end marker. The explorer decodes until it has FileLength − 8 bytes, or until an end marker, so it reads either kind. Bytes after what the decoder used are shown as “After the compressed data”.

Offsets in the decompressed data

Every structure after the header, from the frame size to the last tag, is inside the LZMA data and cannot be pointed at in the file. The reference pages and the explorer count its offsets in the decompressed movie: the file’s first 8 bytes followed by the decoded bytes. Decoded byte n is at offset 8 + n, the frame size is at offset 8, and every offset is the one the same movie would have as an uncompressed FWS file. The 9 bytes at offsets 8 to 16 of the ZWS file are not part of that movie.

In the explorer the file’s own tree shows the 17-byte header and the LZMA data as one range. The decompressed movie is parsed as a separate tree, whose header fields and tags are listed in the model as “offsets in the decompressed data”.

Converting a movie

• To decompress: decode the LZMA data from offset 17 with the 5 property bytes, to FileLength − 8 bytes, then write FWS, the version byte, the FileLength and the decoded bytes. The version stays as it was.
• To hand the body to a standard LZMA tool, make it a .lzma (“LZMA alone”) stream: the 5 property bytes, then FileLength − 8 as a 64-bit little-endian number, then the LZMA data. ZWS has no 8-byte size, so CompressedLength is dropped and the size inserted.
• To compress: take an uncompressed SWF 13 or later movie, compress everything after its first 8 bytes with LZMA, and write ZWS, the version, the unchanged FileLength, the length of the compressed data, the 5 property bytes and the data.

The explorer’s decoder was checked against .lzma streams made by xz --format=lzma from 2.5 MB of text, with and without a known size and cut short: its output matched byte for byte.