Before a writer puts modified (dirty) data into a primary file it stores that data in a transaction log file. If the write to the log fails, the primary file is still consistent; if the write to the primary file fails, the log holds enough to bring the primary file back to a consistent state. This page follows Maxim Suhanov’s Windows registry file format specification.
Old format
Used from Windows NT up to Windows 8 and Windows Server 2012. A log (old format) is a partial base block, a dirty vector and dirty pages.
Base block
The first sector holds a partial copy of the base block, made from data in memory, not from the primary file. Only the first clustering factor × 512 bytes are written. The writer changes it:
- The file type is 1 for a log as of Windows XP, and 2 in Windows 2000 (where 1 was reserved for the
alternate file of the System hive,
SYSTEM.ALT, a mirror of the primary file). - In Windows 2000, XP and Server 2003 the primary sequence number is raised before the log is written and the secondary one after. From Windows Vista, one number, and the same number raised by one, are written together before the dirty data.
- The checksum is calculated again.
A log is valid when its base block is as expected (with these changes) and its two sequence numbers are equal. An invalid log can be repaired by the self-healing process, which resets the base block fields in memory: the signature, the hive bins data size (from the primary file’s size), the clustering factor, the sequence numbers (to 1) and the checksum. A valid log can be applied to a dirty hive when its last written timestamp equals that of the primary file’s base block (or of the first hive bin’s header when that base block has a wrong checksum), and also after self-healing. If the primary file’s base block has a wrong checksum, it is recovered from the log’s base block, with the file type set back to 0.
In Windows 2000, XP and Server 2003 the same memory region is written to both files, so after a successful write the primary file’s sequence numbers are both N and the log’s are both N − 1. From Windows Vista a copy is used for the log, so both files carry the same numbers.
Dirty vector
Starting at the second sector:
| Offset | Length | Field | Meaning |
|---|---|---|---|
| 0 | 4 | Signature | DIRT |
| 4 | … | Bitmap | One bit for each 512-byte page of the hive bins data |
Bit 1 is the first 512-byte page of the hive bins data, bit 2 the second, and so on. Bits are packed into bytes with
bit 1 as the least significant bit of the first byte. A set bit means the page is dirty. The bitmap has
hive bins data size ÷ 512 bits; the state of the base block is not recorded. Windows tracks changes in
4096-byte blocks, so the bytes are expected to be 0x00 or 0xFF. Padding probably follows the
bitmap, up to a sector boundary.
Dirty pages
They start at the sector after the last sector of the dirty vector. Each is 512 bytes, with no gaps. The first belongs to the first set bit, and so on; the count equals the number of set bits, and leftovers of earlier pages may follow. A dirty page goes to file offset 4096 + 512 × bit position, the position counted from zero. During recovery, contiguous pages in one hive bin are handled together and the bin is checked: its signature must be right, its size at least 4096 and its offset equal to that of the bin in the primary file; recovery stops at an invalid bin, and an invalid dirty bin is ignored.
New format
Introduced in Windows 8.1 and Windows Server 2012 R2. A log (new format) is a base block and log entries. The first sector is a modified partial copy of the base block as in the old format, with the file type set to 6. Entries start at the second sector, each at an offset divisible by 512, with a size that is a multiple of 512 and no gaps:
| Offset | Length | Field | Meaning |
|---|---|---|---|
| 0 | 4 | Signature | HvLE |
| 4 | 4 | Size | Size of the entry |
| 8 | 4 | Flags | A partial copy of the base block’s flags when the entry was made; only 0x00000001 is kept |
| 12 | 4 | Sequence number | The possible value of both sequence numbers in memory after this entry is applied |
| 16 | 4 | Hive bins data size | A copy of the base block’s field when the entry was made |
| 20 | 4 | Dirty pages count | Number of pages attached |
| 24 | 8 | Hash-1 | Marvin32 hash, below |
| 32 | 8 | Hash-2 | Marvin32 hash, below |
| 40 | … | Dirty pages references | 8 bytes each: offset of the page from the start of the hive bins data (4), size of the page (4) |
| … | … | Dirty pages | In the order of the references, with no alignment or gaps |
Hash-1 is the Marvin32 hash of the data from the start of the first page reference, Size − 40 bytes long.
Hash-2 is the Marvin32 hash of the first 32 bytes of the entry, including Hash-1. Both use the seed
82 EF 4D 88 7A 4E 55 C5. Written as a 64-bit number most significant byte first, that is
0x82EF4D887A4E55C5, and the low 32 bits (0x7A4E55C5) are the first word of the Marvin32
state and the high 32 bits (0x82EF4D88) the second; the 8-byte hash fields are the two words of the
result, first word first, stored little-endian. This reading reproduced the stored hashes of the log entries in the
yarp project’s Windows 8.1 test logs.
Notes on use:
- A log may hold several entries written together, and old entries that were already applied; its base block is updated only when the first entry is written.
- If the primary file is dirty and has a valid checksum, only subsequent entries are applied: those with a sequence number equal to or greater than the primary sequence number of the log’s base block. If its checksum is wrong, its base block is recovered from the log first.
- If an entry with sequence number N is not followed by one with N + 1, recovery stops after N. If the first entry does not have the expected number (the log’s primary sequence number, not less than the secondary sequence number of the primary file’s valid base block), recovery stops.
- An entry with a wrong Hash-1, Hash-2 or hive bins data size (not a multiple of 4096) stops recovery: only the earlier entries are applied.
- The primary file is grown to the hive bins data size of the entry being applied. Dirty hive bins are checked; an invalid one is replaced by a dummy bin and recovery goes on.
Dirty state of a hive
A hive is dirty, and needs recovery, when the base block of its primary file has a wrong checksum or its primary sequence number differs from its secondary one. If it is not dirty but a log (new format) holds subsequent entries, they are ignored.
More than one log file
A writer uses no log, one log (.LOG) or two (.LOG1 and .LOG2), the dual-logging
scheme of Windows Vista. In that case an empty third file .LOG may be left from an installation image; with
a single log, two empty files of the dual scheme may be present.
Old format
Normally only .LOG1 is used. After an error writing the primary file the writer switches to
.LOG2, which then holds a cumulative log of the pages that were not written in full and those dirtied since,
and while the errors persist it swaps the file on every attempt, so a consistent copy of the earlier data stays in the
other file if a crash damages the current one. After a successful write, .LOG1 is used again.
Normally the first log recovers a dirty hive. Before Windows 8, the second is used only if the primary file’s base block
is invalid, the first log has no valid base block copy or one with a different last written timestamp, and the second log
has a valid copy with a different timestamp and a more recent log of dirty data. Windows 8 uses the second log even when the
primary file’s base block is valid and even when the timestamp of its base block copy matches. A kernel update for
Windows 7 (6.1.7601.19160, KB3140410, March 2016) works like Windows 8. A boot loader uses the log whose base block copy has a
matching timestamp, trying .LOG1 first. The sequence numbers in a log do not record a mid-update state, so after a
crash while a log was written there is no sure sign of which log is inconsistent.
New format
The writer swaps the log file regularly, so entries can be split between the two files and the first file does not hold the earlier entries for certain. If the primary file’s base block is valid, entries from both logs are applied, the log with the earlier entries first; if recovery stops, it continues with the next log, whose first entry must have the sequence number N + 1 after the last applied N. If the base block is invalid, only the log with the latest entries is used.
Flushing, sector size and clustering
With the old log format a flush ends when the data is in the primary file; with the new one it ends when the data is in a log, and writing to the primary file may be delayed by up to an hour, so dirty data stays unreconciled in the meantime. As of Windows 8 the clustering factor is always 1 and the sector size is taken as 512 bytes, so a base block copy in a log is 512 bytes whatever the disk. Windows supports logical sectors of 512 and 4096 bytes only (4096 from Windows 8 and Server 2012).