.TXT File Format

Introduction

UTF-8 is the usual encoding of text today. It stores each Unicode character in one to four bytes. The 128 ASCII characters keep their one-byte values, 00 to 7F, so a file of pure ASCII is already a UTF-8 file, and every byte from 80 up is part of a character beyond ASCII. Ken Thompson and Rob Pike designed it in 1992 for the Plan 9 operating system. It is defined today by RFC 3629 and by the Unicode Standard.

A UTF-8 text file has no header and no length fields: an optional byte order mark, then characters to the end of the file, split into lines by line-ending characters. Nothing in the file names its encoding except that byte order mark, and most UTF-8 files have none. A reader recognises UTF-8 by its bytes instead: the multi-byte sequences follow strict rules that text in other encodings almost never meets by chance.

Every character above 7F is stored differently from UTF-16, UTF-32 and the old code pages, so the same bytes mean different text in each. That makes UTF-8 a format of its own, described here as one.

A file dropped on any page of this site opens in the explorer, which shows its structure (the byte order mark, each line, and for a line beyond ASCII each character with its bytes) and a preview of the text.

Structure

The file has up to four parts, in this order. Only the lines are always there.

Byte order mark

The character U+FEFF, ZERO WIDTH NO-BREAK SPACE, written at the very start of a file, is the byte order mark (BOM). In UTF-8 it is the three bytes EF BB BF. UTF-8 is a sequence of single bytes, so it has no byte order to mark; here the mark only works as a signature that says the text is UTF-8. It is not part of the text, and a reader drops it.

It is optional. The Unicode Standard allows it but does not require it. The Unicode FAQ says it might be useful to identify the data as UTF-8, but that some receivers do not expect it. Programs that look for ASCII at the start of a file can stumble on it: a Unix script whose first line must start with #!, for one. Most UTF-8 files have no BOM; among the files examined for this site only a small share had one.

The same character starts a file in each Unicode encoding, written in that encoding’s own bytes, so the first bytes tell the encodings apart:

A U+FEFF further on in the text (EF BB BF in the middle of a file) is usually a byte order mark left behind where two files were joined. The explorer reports it as a problem.

00000000  EF BB BF 63 61 66 C3 A9 0D 0A                 ...caf....

Encoding

Each Unicode code point, from U+0000 to U+10FFFF, is written as one to four bytes. The first byte, the lead byte, says by its high bits how many bytes the character takes; every byte after it is a continuation byte, 10xxxxxx, carrying six bits of the code point. This table is the whole of valid UTF-8 (RFC 3629, and table 3-7 of the Unicode Standard, “Well-Formed UTF-8 Byte Sequences”):

The narrower ranges after E0, ED, F0 and F4 rule out the forms that are not allowed (see Problems): a character written in more bytes than it needs, a UTF-16 surrogate, and anything above U+10FFFF. The bytes C0, C1 and F5 to FF never appear in valid UTF-8, and 80 to BF appear only as continuation bytes.

00000000  43 61 66 C3 A9 20 E2 82 AC 34 0A E4 B8 AD E6 96  Caf.. ...4......
00000010  87 20 F0 9F 98 80 0A                             . .....

Lines

A line is the characters up to a line ending. The explorer counts one line for each line ending, plus one for any text after the last, so a␊b␊ and a␊b are both two lines. Lines are numbered from 1, as editors number them.

POSIX defines a line as zero or more characters followed by a newline, so a file whose last line has no line ending ends, strictly, in an incomplete line. A few tools notice: diff prints “No newline at end of file”, and wc -l, which counts line feeds, does not count that line. It is common and harmless, and the explorer gives it as a note, not a problem. In the files examined for this site about one in five had no line ending after the last line.

Characters are counted as code points, not bytes, and not counting the line endings; an invalid sequence counts as one. The longest line is measured in characters.

The tab (09) and the form feed (0C, a page break, as in the text of RFCs) are ordinary in text. The explorer counts them and does not report them.

Line endings

Unicode’s newline guidelines (chapter 5 of the Standard) list CR, LF, CR LF and NEL as newline functions, and LS and PS as separators that mean only one thing. The explorer ends a line at any of the six:

A CR directly followed by an LF is one CR LF ending, not two. The line-breaking rules of UAX #14 also give the vertical tab and the form feed a mandatory break; the explorer does not split lines at them, as editors do not.

In the files examined for this site LF was the commonest ending, CR LF next; CR alone was rare. A file with more than one kind of ending is reported as having mixed line endings, with the first line that differs from the commonest kind. It is usually a file edited on two systems, or CR LF lines with Unix lines appended. A lone CR among CR LF lines is common in saved console output, where a progress line was redrawn with a CR.

End of the file

The text simply stops at the end of the file. Whether the last line has a line ending is covered under Lines. Two other things can follow the text:

• Ctrl-Z. CP/M stored file sizes in 128-byte records, so it marked the end of the text with the byte 1A (SUB, Ctrl-Z), and MS-DOS programs kept writing it. The explorer shows trailing 1A bytes as an End-of-file mark. A 1A inside the text is a control character.
• Zero padding. NUL bytes after the text, left by programs that preallocate a file or write it in fixed-size blocks. The explorer shows them as Zero padding and reports them. A file that is mostly such zeros is not taken as text.

Problems

A byte sequence outside the table under Encoding is not UTF-8. These are the ways it goes wrong:

• A stray continuation byte, 80 to BF with no lead byte before it.
• A lead byte without its continuation bytes, such as C3 followed by an ASCII letter. This is the usual sign of text in another encoding: é in Windows-1252 is the single byte E9, a lead byte that wants two more.
• Bytes that UTF-8 never uses: C0, C1, F5 to FF.
• An overlong form: a character written in more bytes than it needs, such as C0 80 for U+0000 or E0 80 AF for /. They are banned because a check for / or NUL could be slipped past with them. Java’s “modified UTF-8”, in its DataInput and DataOutput classes, writes U+0000 as C0 80.
• An encoded surrogate, ED A0 80 to ED BF BF (U+D800 to U+DFFF). CESU-8 (UTR #26) writes a character above U+FFFF as two such three-byte surrogates, six bytes in all; the Unicode FAQ calls this not conformant to UTF-8. WTF-8 allows unpaired surrogates, to carry Windows file names.
• A value above U+10FFFF, such as F4 90 80 80. RFC 2279 (1998) allowed five- and six-byte forms up to U+7FFFFFFF; RFC 3629 (2003) removed them.
• A character cut short by the end of the file, as when a copy was interrupted.

Telling it apart

Without a byte order mark, UTF-8 is recognised by its rules. Text in an old code page that has accented letters breaks them almost at once, and binary data breaks them within a few bytes. The explorer’s parser for this format checks, in order:

1. A file that starts with FF FE or FE FF, a UTF-16 byte order mark, is turned down at once.
2. A file that starts with EF BB BF is always UTF-8. Any invalid sequences in it are then reported, not held against it.
3. Without the mark, the first 64 KB must not look binary: NUL in more than one character in a hundred, or other control characters in more than one in ten, turns the file down. (Tab, LF, form feed, CR, escape and Ctrl-Z do not count.)
4. Then every byte sequence in the file must be valid UTF-8. The first invalid one turns the file down, and the next formats are tried. Overlong forms and encoded surrogates do not turn it down, and nor does a character cut short by the very end of the file.

UTF-8 is tried first of the text formats, because it is by far the commonest and its test is exact; the legacy code pages, which accept almost any bytes, are tried last. A .txt file that no format accepts, binary data under a text name, is still shown as UTF-8, with its invalid sequences reported.

Among the files examined for this site nearly every .txt file was UTF-8, and most of those were pure ASCII. When text in UTF-8 is read in the wrong code page it turns into “mojibake”: é shown as é is UTF-8 read as Windows-1252.

Variants

• UTF-8 text – one to four bytes a character, ASCII unchanged
• UTF-16LE text – two or four bytes a character, low byte first
• UTF-16BE text – two or four bytes a character, high byte first
• UTF-32LE text – four bytes a character, low byte first
• UTF-32BE text – four bytes a character, high byte first
• Legacy code page text – a pre-Unicode code page that the file does not name

Links