.MOBI File Format – Text records and compression

Trailing entries

When the MOBI header’s extraDataFlags are not 0, every text record ends with extra entries that are not part of the compressed text. They are stored in bit order after the text: the multibyte entry (bit 0) first, then the entry for bit 1, then bit 2 and so on, so the entry of the highest set bit is last. A reader removes them from the end:

1. For each set bit from 15 down to 1: read a backward-encoded integer ending at the last byte of the record. It is the size of that entry, including the integer itself. Remove that many bytes from the end.
2. If bit 0 is set: the last remaining byte is the multibyte entry’s size byte. Its low two bits give n, 0 to 3; remove n + 1 bytes. The n bytes before the size byte are the start of a multibyte character that the next record finishes; the same bytes appear at the start of the next record’s text too. They are not compressed but are encrypted.

The entry for bit 1 is the trailing byte sequence (TBS), which tells a Kindle which table of contents entries start, end or continue in this record, for quick navigation. Its layout varies with the index type and is only partly understood; readers that only show the text skip it. The contents of the bit 2 entry, uncrossable breaks, are not documented.

Variable-width integers

MOBI uses a big-endian variable-width integer in trailing entries and indexes. Each byte carries 7 bits of the value in its low bits; bit 7 marks one end of the number.

• Forward encoded, read from the first byte onwards: bit 7 is set only on the last byte. Used in index entries and CNCX strings.
• Backward encoded, read from the last byte backwards: bit 7 is set only on the first byte. Used for the sizes of trailing entries, which are read from the end of a record.

For example, 0x11111 is 04 22 91 forward and 84 22 11 backward. A value below 128 is one byte with bit 7 set either way: 5 is 85.

PalmDOC compression

Compression type 2 is a byte-oriented LZ77 made for the slow processors of Palm handhelds. Each record decompresses to at most 4,096 bytes, and back references never reach into a previous record. The decoder reads one byte at a time:

A compressor looks back up to 2,047 bytes for a match of 3 or more bytes. Some old PalmDOC tools also XORed every byte with 0xA5 before storing it; a decoder that gets nonsense can try undoing that first.

Example

74 68 65 E3 61 74 F4 80 43

HUFF/CDIC compression

Compression type 17480 (DH) codes the text with a Huffman code whose symbols are phrases from a dictionary. The tables are in the records that the MOBI header’s huffmanRecordOffset and huffmanRecordCount give: one HUFF record, then one or more CDIC records. The method is not documented by Amazon; this description follows the decoders in calibre and KindleUnpack.

HUFF record

The first table is indexed by the next 8 bits of the input. Each 32-bit entry holds: bits 0–4, the code length (1 to 32, never 0); bit 7, a flag that the code is complete within 8 bits (always set when the length is 8 or less); bits 8–31, the maxcode for that length, from which a decoder computes ((maxcode + 1) << (32 − length)) − 1.

The second table holds 32 pairs, one per code length 1 to 32: the smallest code of that length, then the largest. A decoder shifts each left by (32 − length), adding ones below the largest, as for the first table.

CDIC records

At each offset (from byte 16) is a phrase: a UInt16 whose bits 0–14 are its length and whose bit 15 is a flag, then the phrase’s bytes. With the flag set, the bytes are plain text. With it clear, the bytes are themselves Huffman-coded and must be decoded the same way before use; a decoder then keeps the result. The phrases of all CDIC records are numbered in one sequence.

Decoding

1. Read the record as a stream of bits, most significant first, and look at the next 32 bits as an unsigned number, code.
2. Look up the top 8 bits of code in the first table to get a length, the flag and maxcode. If the flag is clear, increase the length while code is less than the second table’s smallest code for that length, then take maxcode from the second table.
3. The phrase number is (maxcode − code) >> (32 − length). Output that phrase and move on by length bits.
4. Stop when fewer bits are left than the next code needs; the end of the record is padding.

The text

Decompressed and joined, the text records make one HTML document in the MOBI header’s encoding. It is the HTML of the source book, simplified, with a few changes:

Only HTML 3.2-level formatting is supported: font, b, i, blockquote, table and the like, with attributes such as width and height in place of CSS. There are no style sheets.