A value too large for a record is kept in the table's long value tree, a B+-tree of its own whose pages have the long value flag (0x80) and whose root the catalog gives (a row of Type 4). The record holds only the value's id.
Reading a long value
- In the record, a tagged column whose flags byte has 0x04 holds an id: four bytes (files with pages up to 8 KiB) or eight (16 and 32 KiB).
- The tree's key for the value is those bytes in reverse order. Find it in the long value tree: this entry is the value's root.
- The root's data is eight bytes: a reference count (32 bits) and the value's size after decompression (32 bits).
- Its segments follow in key order. Each has the key of the root followed by the offset of the segment (32 bits, most significant byte first) and holds the next part of the value. Join the segments in offset order.
- If the joined size is not the size the root gives, the value was stored compressed: decompress it.
The segments of one value may continue on the next leaf page; follow the leaf pages' next-page links.
Compression
A tagged value with the compressed flag (0x02), and a long value stored compressed, starts with one byte whose top five bits say how the rest is compressed:
| Method | First byte | Meaning |
|---|---|---|
| 0 | 0x00 – 0x07 | Not compressed: the data follows the byte. |
| 1 | 0x08 – 0x0f | 7-bit: every byte of the data was below 128 and is packed seven bits to a byte (used for UTF-16 text of ASCII characters too, whose bytes are all below 128). |
| 2 | 0x10 – 0x17 | 7-bit Unicode: every UTF-16 unit was below 128 and is packed seven bits to a unit, and unpacks to a 16-bit unit. |
| 3 | 0x18 – 0x1f | XPRESS: two bytes with the size after decompression, then a stream of Plain LZ77. |
| 4, 5, 6 | 0x20 – 0x37 | Scrub, XPRESS9 and XPRESS10: not decoded. |
The low three bits of the first byte are not documented.
7-bit packing
The values are read least significant bit first: each byte supplies its eight bits to a bit stream, and every seven bits of the stream are one value. The stream has no length: the padding in the last byte can make a trailing zero value, which is dropped, and the explorer keeps a UTF-16 text's last unit whole.
XPRESS
Plain LZ77 as in Microsoft's [MS-XCA]: 32-bit flag words, most significant bit first; a clear bit is a literal byte, a set bit a match. A match is a 16-bit word, the top 13 bits the distance minus one and the low three the length minus three; a length field of 7 continues in a four-bit nibble (shared by two consecutive matches), then a byte, then a 16-bit and a 32-bit number. The decoder stops at the size in the header.
Links
Reference
- Extensible Storage Engine (ESE) Database File (EDB) format – long values and compressed data
- MS-XCA – the XPRESS algorithms