.TXT File Format

Introduction

UTF-32BE stores each Unicode character as one 32-bit number, its code point, written high byte first (big-endian). Every character takes exactly four bytes, so the hundredth character of a file is always at byte 396, or 400 after a byte order mark. Read in a hex dump, each unit shows its code point directly: 00 00 20 AC is U+20AC. It is the simplest Unicode encoding to work with, and the largest: ASCII text takes four times the space it takes in UTF-8. UTF-32 is used inside programs far more than in files, and UTF-32BE files are rarer still; none of the text files examined for this site used it.

UTF-32 was defined in Unicode Standard Annex #19, and has been part of the core Unicode Standard since version 4.0 (2003). It is the part of ISO 10646’s UCS-4 that Unicode uses, the code points from 0 to 10FFFF. Browsers do not decode it: it is not in the WHATWG Encoding Standard, whose list of encodings is closed. The explorer decodes it itself.

A UTF-32BE text file has no header and no length fields: an optional byte order mark, then 4-byte units to the end of the file. The same text in UTF-32LE has every unit’s bytes reversed, and in UTF-16BE or UTF-8 it has different bytes altogether, so each is a format of its own.

A file dropped on any page of this site opens in the explorer, which shows its structure (the byte order mark, each line, and for a line beyond ASCII each character with its bytes) and a preview of the text.

Structure

The file has two parts. Its length is a whole number of 4-byte units.

Byte order mark

The character U+FEFF, ZERO WIDTH NO-BREAK SPACE, written at the very start of a file, is the byte order mark (BOM). As a 32-bit number high byte first, it is 00 00 FE FF. It is not part of the text, and a reader drops it. Unlike UTF-32LE’s mark, it is hard to mistake for UTF-16’s: a UTF-16 file starts with two zero bytes only if its text starts with U+0000.

Without a mark the Unicode Standard reads unmarked “UTF-32” as big-endian, as it does UTF-16, so a file without one is UTF-32BE by default. The explorer still checks the bytes rather than assume it.

A U+FEFF further on in the text is usually a byte order mark left behind where two files were joined, and the explorer reports it.

Encoding

Each character is one 32-bit unsigned number, the code point itself, written in four bytes from the highest to the lowest. There is nothing to compute. A unit is valid when its value is a Unicode scalar value:

So the first byte of every unit is 00, and the second is 00 to 10.

00000000  00 00 FE FF 00 00 00 43 00 00 00 61 00 00 00 66  .......C...a...f
00000010  00 00 00 E9 00 00 00 20 00 00 20 AC 00 00 00 34  ....... .. ....4
00000020  00 00 00 0A 00 01 F6 00 00 00 00 0A              ............

Lines

A line is the characters up to a line ending. The explorer counts one line for each line ending, plus one for any text after the last, so a␊b␊ and a␊b are both two lines. Lines are numbered from 1, as editors number them.

POSIX defines a line as ending in a newline, so a last line with no line ending is, strictly, incomplete. It is common and harmless, and the explorer gives it as a note, not a problem.

Characters are counted as units, one per character, not counting the line endings. The longest line is measured in characters. Tabs (00 00 00 09) and form feeds (00 00 00 0C, page breaks) are counted and not reported.

Line endings

Each line-ending character is one unit, four bytes, high byte first. Unicode’s newline guidelines (chapter 5 of the Standard) list CR, LF, CR LF and NEL as newline functions, and LS and PS as separators; the explorer ends a line at any of the six:

A file with more than one kind of ending is reported as having mixed line endings, with the first line that differs from the commonest kind. The vertical tab and the form feed do not end lines.

End of the file

The text stops at the end of the file. There is no end-of-file mark in UTF-32 files: the Ctrl-Z of MS-DOS belongs to byte-oriented text. Zero units at the end (00 00 00 00) are NUL characters, and the explorer reports them as such.

When the file’s length is not a multiple of four, the last one to three bytes are part of a unit. The explorer reports them as a character cut short by the end of the file.

Problems

A unit is wrong when its value is not a Unicode scalar value:

• Above 10FFFF, such as 00 11 00 00 (110000). This is an invalid byte sequence. It is also what text in another encoding looks like when read as UTF-32.
• A surrogate, D800 to DFFF, such as 00 00 D8 3D. Surrogates belong to UTF-16 only; one here is usually UTF-16 converted unit by unit without joining the pairs. The explorer reports it as an unpaired surrogate.
• A unit cut short by the end of the file.

A strict decoder stops with an error; a lenient one puts U+FFFD, the replacement character �, in place of each bad unit. The explorer shows the bad unit’s bytes in the tree, reports it, and shows U+FFFD for it in the text preview.

Telling it apart

UTF-32 is the easiest Unicode encoding to recognise without a byte order mark, because almost no other data passes its test. The explorer’s parser for this format checks:

1. With the mark 00 00 FE FF: the units after it, up to 64 KB, are read. If more than one in ten are not code points, the file is turned down.
2. Without it: every 4-byte unit of the first 64 KB, read high byte first, must be a code point (at most 10FFFF, not a surrogate). A single bad unit turns the file down. Text in UTF-16 or in an 8-bit encoding never passes: its bytes make values far above 10FFFF. Nor does UTF-32LE: read high byte first, its units are far above 10FFFF too.
3. The units must not look binary: NUL in more than one character in a hundred, or other control characters in more than one in ten, turns the file down.

The text formats are tried UTF-8 first, then UTF-32LE, then this one, then the two UTF-16 formats.

Variants

• UTF-8 text – one to four bytes a character, ASCII unchanged
• UTF-16LE text – two or four bytes a character, low byte first
• UTF-16BE text – two or four bytes a character, high byte first
• UTF-32LE text – four bytes a character, low byte first
• UTF-32BE text – four bytes a character, high byte first
• Legacy code page text – a pre-Unicode code page that the file does not name

Links