Introduction
The Extensible Storage Engine (ESE, also called JET Blue) is the database engine inside
Microsoft Windows (esent.dll) and Exchange Server. Programs that need a fast
local store with transactions use it, and its files turn up all over a Windows machine:
the Windows Search index (Windows.edb), the Active Directory database
(NTDS.dit), the Internet Explorer and Edge cache (WebCacheV01.dat),
the System Resource Usage Monitor (SRUDB.dat), the Windows Update store
(DataStore.edb), certificate services and Exchange mailbox databases
(.edb). It has nothing to do with the Jet engine of Microsoft Access
(.mdb), which is a different format.
A database is one file of equal-sized pages that hold tables, indexes and long values as
B+-trees. Beside it are the transaction log files (.log) and a checkpoint file
(.chk) that hold changes not yet written to the database; a database whose state
says dirty shutdown is not complete without them. Windows keeps these files locked
while it runs, so they are normally copied from a shadow copy or an image.
Drop a database on this site and the explorer shows its structure, every page and every byte of it, and its tables with their columns and rows. Large databases open page by page: nothing reads more of the file than the page, table or row window on the screen.
Reading a database
A reader needs seven operations. Each is described, with its numbers, on the page named.
- Open. Read the 668-byte file header: check the signature
EF CD AB 89at offset 4, and take the page size (offset 236), the format revision (232) and the state (52). Page number n starts at byte (n + 1) × page size; pages 0 and 1 of the file are the header and its shadow copy. - Check a page. Recompute its checksums and compare them with the page header. Pages of 16 and 32 KiB carry four.
- Walk a tree. From a tree's root page, follow the first child of every branch page down to the first leaf and then the next page link of each leaf (B+-trees). To find one key, descend by the keys in the branches.
- Read the catalog. The table
MSysObjectshas its root at page 4. Its rows describe every table, column, index and long value tree and give the root page of each (the catalog). - Read a row. Each leaf entry of a table is a record: take its fixed, variable and tagged columns, using the catalog's column definitions, and honour the NULL bitmap.
- Follow a long value. A tagged column flagged as a long value holds an id; its bytes are in the table's long value tree under that id (long values), in segments, and are decompressed if they were stored compressed.
- Count and measure. The space trees list the pages a tree owns and the ones still free.
Structure
- Page 0 – the file header: 668 bytes, then zeros to the end of the page
- Page 1 – a copy of the file header (the shadow header)
- Page 1 of the database (at byte 2 × page size) – the database root; pages 2 and 3 its space trees
- Page 4 – the root of the catalog,
MSysObjects; its backup copyMSysObjectsShadowis at page 24 - Every other page – a page of a table, an index, a long value tree or a space tree, each page a leaf, a branch or a root
Every page has a header (40 bytes, or 80 with pages of 16 and 32 KiB), values that grow up from it, and a table of four-byte tags at the end of the page that grows down and says where each value is. A tree's root page also holds the tree's space information.
Versions
The format version in the header is always 0x620. What changes is the format revision, which decides details of the
page header, the tags and the records:
| Revision | Introduced with | What changed |
|---|---|---|
| 0x00 – 0x09 | Early Exchange and Windows 2000 to XP | XOR checksum seeded with 0x89abcdef; the page number is in the page; no ECC. |
| 0x0b | Exchange 2003 SP1 (Vista is 0x0c) | ECC checksum next to the XOR checksum; the XOR is seeded with the page number. |
| 0x11 | Windows 7 | 2, 16 and 32 KiB pages, an extended page header on the large pages, compressed columns. |
| 0x14 and up | Exchange 2013 and Active Directory of Windows Server 2016 (0x14), Windows 8 to 11 (0x1e and up) | More catalog columns. In the files with 16 and 32 KiB pages every tagged value starts with a flags byte and long value ids are 8 bytes. |
The pages say which revision a file has, but the explorer decides the large-page details from the page size, as the files it was checked against require.
Not described
Known, not yet covered
- Viewer: index keys. The keys in an index are normalised byte strings (integers with the sign bit flipped, text as collation weights); the explorer shows them as hexadecimal and does not turn them back into column values.
- Viewer: repair. The ECC checksum of a page can correct a single flipped bit; the explorer verifies it and reports a failure but does not correct the page.
- Reference and viewer: the revisions before 0x0b (an XOR checksum seeded with 0x89abcdef and the page number in the page). They are read as libesedb describes them; no real file of those revisions was available to check them.
- Reference and viewer: multi-values stored with the 0x10 flag (the first 16 bits give the size of the first of two values). Written from libesedb's description; no file checked held one.
Unknown
- Reference and viewer: the transaction log files (
.log), the checkpoint file (.chk) and the streaming file's pages: their formats are not published, and the explorer does not open the logs (a streaming file shows its two header pages and then bytes). - Reference and viewer: the compression methods XPRESS9 and XPRESS10 (the method number in the top five bits of a compressed value's first byte is 5 and 6) and the scrub fill: named, not decoded.
- Reference and viewer: columns encrypted by Exchange, and several page header and file header fields that libesedb itself lists as unknown (the page flags 0x1000, 0x8000 and 0x10000, bytes 40 to 79 of a large page header apart from the checksums and the page number, and the file header's unknown words). They are shown labelled, as raw numbers.
Related formats
- SQLite database – another page-based database with B-tree tables, in a single file
- Windows thumbnail cache – the thumbnails of Windows Explorer; the Windows Search index names the file each one shows
Links
Software
- esentutl – Microsoft's tool that dumps headers and checks, repairs and defragments a database (Windows)
- ESEDatabaseView – NirSoft's viewer for ESE databases (Windows)
- libesedb tools –
esedbinfoandesedbexportlist and export tables to text (Linux, macOS, Windows)
Developers
- libesedb – Joachim Metz's library that reads ESE databases (C, with Python bindings)
- Extensible Storage Engine API – Microsoft's documentation of the API that creates and reads the databases (C)
Reference
- Extensible Storage Engine (ESE) Database File (EDB) format – libesedb's description of the file format
- MS-XCA – Microsoft's description of the XPRESS compression that long values use