.TXT File Format

Introduction

UTF-32LE stores each Unicode character as one 32-bit number, its code point, written low byte first (little-endian). Every character takes exactly four bytes, so the hundredth character of a file is always at byte 396, or 400 after a byte order mark. That makes it the simplest Unicode encoding to work with, and the largest: ASCII text takes four times the space it takes in UTF-8. It is used inside programs more than in files; the C type wchar_t is 32 bits wide on Linux and macOS, for one.

UTF-32 was defined in Unicode Standard Annex #19, and has been part of the core Unicode Standard since version 4.0 (2003). It is the part of ISO 10646’s UCS-4 that Unicode uses, the code points from 0 to 10FFFF. Browsers do not decode it: it is not in the WHATWG Encoding Standard, whose list of encodings is closed. The explorer decodes it itself.

A UTF-32LE text file has no header and no length fields: an optional byte order mark, then 4-byte units to the end of the file. The same text in UTF-32BE has every unit’s bytes reversed, and in UTF-16LE or UTF-8 it has different bytes altogether, so each is a format of its own. Among the files examined for this site only two were UTF-32LE: JSON files, named .txt, written by a cloud storage program without a byte order mark.

A file dropped on any page of this site opens in the explorer, which shows its structure (the byte order mark, each line, and for a line beyond ASCII each character with its bytes) and a preview of the text.

Structure

The file has two parts. Its length is a whole number of 4-byte units.

Byte order mark

The character U+FEFF, ZERO WIDTH NO-BREAK SPACE, written at the very start of a file, is the byte order mark (BOM). As a 32-bit number low byte first, it is FF FE 00 00. It is not part of the text, and a reader drops it.

These four bytes are also UTF-16LE’s byte order mark, FF FE, followed by the character U+0000 (00 00). Text rarely starts with a NUL, so UTF-32LE is the likelier reading, but the bytes after the mark settle it. The explorer tries UTF-32LE before UTF-16LE and reads the units after the mark: if more than one in ten of them are not code points (above 10FFFF, or surrogates), the file is not UTF-32LE, and UTF-16LE takes it. UTF-16 text read in 4-byte units gives values far above 10FFFF, so it fails this at once.

Without a mark the byte order is not stated; the Unicode Standard reads unmarked “UTF-32” as big-endian, as it does UTF-16. A U+FEFF further on in the text is usually a byte order mark left behind where two files were joined, and the explorer reports it.

Encoding

Each character is one 32-bit unsigned number, the code point itself, written in four bytes from the lowest to the highest. There is nothing to compute. A unit is valid when its value is a Unicode scalar value:

So the fourth byte of every unit is 00, and the third is 00 to 10.

00000000  FF FE 00 00 43 00 00 00 61 00 00 00 66 00 00 00  ....C...a...f...
00000010  E9 00 00 00 20 00 00 00 AC 20 00 00 34 00 00 00  .... .... ..4...
00000020  0A 00 00 00 00 F6 01 00 0A 00 00 00              ............

Lines

A line is the characters up to a line ending. The explorer counts one line for each line ending, plus one for any text after the last, so a␊b␊ and a␊b are both two lines. Lines are numbered from 1, as editors number them.

POSIX defines a line as ending in a newline, so a last line with no line ending is, strictly, incomplete. It is common and harmless, and the explorer gives it as a note, not a problem.

Characters are counted as units, one per character, not counting the line endings. The longest line is measured in characters. Tabs (09 00 00 00) and form feeds (0C 00 00 00, page breaks) are counted and not reported.

Line endings

Each line-ending character is one unit, four bytes, low byte first. Unicode’s newline guidelines (chapter 5 of the Standard) list CR, LF, CR LF and NEL as newline functions, and LS and PS as separators; the explorer ends a line at any of the six:

A file with more than one kind of ending is reported as having mixed line endings, with the first line that differs from the commonest kind. The vertical tab and the form feed do not end lines.

End of the file

The text stops at the end of the file. There is no end-of-file mark in UTF-32 files: the Ctrl-Z of MS-DOS belongs to byte-oriented text. Zero units at the end (00 00 00 00) are NUL characters, and the explorer reports them as such.

When the file’s length is not a multiple of four, the last one to three bytes are part of a unit. The explorer reports them as a character cut short by the end of the file.

Problems

A unit is wrong when its value is not a Unicode scalar value:

• Above 10FFFF, such as 00 00 11 00 (110000). This is an invalid byte sequence. It is also what text in another encoding looks like when read as UTF-32.
• A surrogate, D800 to DFFF, such as 3D D8 00 00. Surrogates belong to UTF-16 only; one here is usually UTF-16 converted unit by unit without joining the pairs. The explorer reports it as an unpaired surrogate.
• A unit cut short by the end of the file.

A strict decoder stops with an error; a lenient one puts U+FFFD, the replacement character �, in place of each bad unit. The explorer shows the bad unit’s bytes in the tree, reports it, and shows U+FFFD for it in the text preview.

Telling it apart

UTF-32 is the easiest Unicode encoding to recognise without a byte order mark, because almost no other data passes its test. The explorer’s parser for this format checks:

1. With the mark FF FE 00 00: the units after it, up to 64 KB, are read. If more than one in ten are not code points, the file is turned down and goes to UTF-16LE.
2. Without it: every 4-byte unit of the first 64 KB, read low byte first, must be a code point (at most 10FFFF, not a surrogate). A single bad unit turns the file down. Text in UTF-16 or in an 8-bit encoding never passes: its bytes make values far above 10FFFF.
3. The units must not look binary: NUL in more than one character in a hundred, or other control characters in more than one in ten, turns the file down.

UTF-32 is tried before UTF-16 because of the shared FF FE 00 00, and after UTF-8, which is far commoner.

Variants

• UTF-8 text – one to four bytes a character, ASCII unchanged
• UTF-16LE text – two or four bytes a character, low byte first
• UTF-16BE text – two or four bytes a character, high byte first
• UTF-32LE text – four bytes a character, low byte first
• UTF-32BE text – four bytes a character, high byte first
• Legacy code page text – a pre-Unicode code page that the file does not name

Links