.MBOX File Format – Index and summary files

Thunderbird summary files (.msf)

Thunderbird keeps a folder's message database, its summary file, in a file with the extension .msf (nsMsgDatabase.cpp calls it the "MSF file"). It is a text file in the Mork format (the Mork code is what opens it) with message-header rows and thread rows, so that a folder can be shown without reading the mail store. The code deletes the file when its version differs from the current one, when a forceReparse property is set (for example when the key limit is reached) or when it is corrupt, and the folder is then reparsed; it does not delete it when Mork cannot open it because of too many open files.

The file starts with the line // <!-- <mdb:mork:z v="1.4"/> --> (morkWriter_kFileHeader).

The Mork text format

A Mork file is a stream of these constructs. The grammar is given in comments in morkParser.cpp with zm:S for optional white space.

Dict      ::= S? '<' DictItem* S? '>'
DictItem  ::= MetaDict | Alias
MetaDict  ::= S? '<' S? Cell* S? '>'          ; meta attributes
Alias     ::= S? '(' ('#')? Hex+ S? Value ')'
Value     ::= '=' ([^)$\] | '\' NonCRLF | Continue | Dollar)*
Table     ::= S? '{' S? Id TableItem* S? '}'
TableItem ::= MetaTable | RowRef | Row
MetaTable ::= S? '{' S? Cell* S? '}'
Row       ::= S? '[' S? Id RowItem* S? ']'
RowItem   ::= MetaRow | Cell
MetaRow   ::= S? '[' S? Cell* S? ']'
Cell      ::= S? '(' Column S? Slot? ')'
Group     ::= GroupStart Content GroupEnd         ; a transaction
GroupStart  ::= S? '@$${' Hex+ '{@'
GroupCommit ::= S? '@$$}' Hex+ '}@'               ; same id as the start
GroupAbort  ::= S? '@$$}~~}@'
Hex       ::= [0-9a-fA-F]

• Ids. An id is a hexadecimal number, optionally followed by : and a scope, which is either a name or ^ and the hexadecimal id of a dictionary entry (1:^80, 8F:ns:msg:db:row:scope:msgs:all style). Row, table and dictionary entries are identified by id within a scope.
• Dictionary (< ... >): aliases (8D=text) that give an id to a string; the id can then be used in place of the string. This is how column names and repeated values (sender names, for example) are stored once. A dictionary form meta cell (f=...) sets the text encoding form of the entries.
• Cell (column slot): the column is a name or ^id of a dictionary entry; the slot is =value (the value text) or ^id (a dictionary entry as the value, as in (^83^85)). The slot letters r, t, " and ' are reserved and give an error ("cell syntax not yet supported").
• Value text: the value ends at an unescaped ). $ followed by two hexadecimal digits is one byte; a backslash escapes the next character, and a backslash before a line break continues the value on the next line (the break is dropped).
• Rows and tables. A table { id ... } lists rows, either by reference (just the id) or in full; a row [ id (cell)(cell) ] may start with - right after [, which the parser takes as "cut all the row's columns"; meta rows and meta tables carry attributes.
• Groups @$${id{@ ... @$$}id}@ wrap a transaction: a commit ends with the same id, an abort with @$$}~~}@ (the parser has separate group-commit and group-abort events).
• Comments and white space: // ... to the end of line and nested /* ... */ are skipped wherever white space is allowed, as is a \ at the end of a line.
• Character classes (morkCh.h): white space, digits, hex, value characters, and the characters that can start (N) and continue (M) a name.

What Thunderbird stores

Names used by nsMsgDatabase.cpp: row scope ns:msg:db:row:scope:msgs:all and table kind ns:msg:db:table:kind:msgs for message headers; row scope ns:msg:db:row:scope:threads:all with kinds ns:msg:db:table:kind:thread and ns:msg:db:table:kind:allthreads for threads. Numbers are written as lower-case hexadecimal text (%lx, %llx for 64-bit values). Columns:

The meaning of the flags bits is in the status headers section. The position of each message in the mbox is not one of the columns above in a current file (msgOffset is marked obsolete); where Thunderbird gets the position from is not described here.

Mutt header caches

Mutt's header cache is not used for mbox folders. Its manual says header caching is supported for IMAP, POP, Maildir and MH folders only (an mbox is read from one file, so there is nothing to speed up). It is an optional feature built on one of the key-value database libraries Tokyo Cabinet, Kyoto Cabinet, tkrzw, LMDB, QDBM, GDBM or Berkeley DB. The format, from hcache.c, is therefore that of the library plus Mutt's value layout:

• Files. $header_cache names a file (one database for all folders) or a directory (one database per folder). For Maildir and MH the file inside a directory is named after the 32 hexadecimal digits of the MD5 sum of the folder name (and -charset appended when Mutt is built without iconv); for IMAP and POP the caches are .hcache files in the cache directory tree.
• Value. Each cached message is one record. Integers are 4 bytes in the machine's own byte order (memcpy). Strings are a 4-byte length (including the terminating NUL; 0 means a null pointer) and the bytes, converted to UTF-8 when the user's charset is not UTF-8. A record is: a validity field (a struct timeval or the IMAP UIDVALIDITY, whichever the cache uses), a 4-byte version number (see below), the raw bytes of Mutt's in-memory HEADER structure with its transient fields cleared, then the envelope, the top MIME body and the Maildir flags string.
• Envelope. In order: the address lists return_path, from, to, cc, bcc, sender, reply_to, mail_followup_to (each a 4-byte count, then per address the string personal, the string mailbox and a 4-byte group flag; with exact-address builds also the original text first), the strings list_post and subject, a 4-byte offset of the "real" subject inside the subject (or −1), the strings message_id, supersedes, date, x_label, the spam buffer (flag 1 or 0, then data, used offset and size), and the string lists references, in_reply_to and userhdrs (a 4-byte count and the strings).
• Body. The raw bytes of the BODY structure (pointers cleared), the strings xtype and subtype, the parameter list (count, then attribute and value strings), and the strings description, form_name, filename, d_filename.
• Version check. The version number is the first 4 bytes of an MD5 digest of a compiled-in structure-layout constant mixed with the user's spam and nospam patterns. A record whose version differs is discarded and refetched; a database that cannot be opened is deleted and recreated. So a cache is valid only for the Mutt build and settings that wrote it; because the HEADER and BODY structures are copied as memory, it is also specific to the machine's architecture and byte order.

Other programs keep their own caches or databases beside mbox files; none of them is described here.