.TXT File Format

Introduction

Before Unicode, each system and each group of languages had its own code page: a table that gives each byte value a character. The single-byte code pages have 256 characters. Their first half, 00 to 7F, is ASCII, and the second half holds the letters and symbols of one region: Windows-1252 and ISO 8859-1 for Western Europe, Mac OS Roman on the classic Mac, code page 437 on the IBM PC, Windows-1251, KOI8-R and code page 866 for Russian. Chinese, Japanese and Korean need thousands of characters, so their code pages use two bytes for each one beyond ASCII: GBK, Big5, Shift_JIS, EUC-JP and EUC-KR. Windows calls both kinds “ANSI code pages”, and Notepad still saves in the computer’s own code page under the name “ANSI”.

Strictly, each code page is a format of its own: the byte E9 is é in Windows-1252, È in Mac OS Roman, Θ in code page 437 and й in Windows-1251. But nothing in a text file says which one was used. There is no byte order mark and no header, and a single-byte code page accepts every byte, so the file reads without error in all of them. This page therefore describes them together, as one format, and the explorer guesses which code page a file uses.

The code page shown is a guess. The explorer says so each time, gives its reasons, and shows what the commonest bytes would be in each candidate, so that a reader can judge. No program can be sure, and with only a few accented letters in a file a guess is often wrong.

A file dropped on any page of this site opens in the explorer, which shows its structure (each line, and the bytes of any character beyond ASCII), the guessed code page with its reasons, and a preview of the text in that code page.

Structure

The file has up to three parts, in this order. Only the lines are always there.

Byte order mark

Code pages have no byte order mark, and nothing else in the file names the code page. Formats built on text can say it, as HTML does with <meta charset> and email with its Content-Type header, but a plain text file cannot.

A file that starts with a Unicode byte order mark is one of the Unicode formats, not this one: EF BB BF UTF-8, FF FE UTF-16LE, FE FF UTF-16BE, FF FE 00 00 UTF-32LE and 00 00 FE FF UTF-32BE.

One exception comes here. GB18030, China’s national encoding, writes U+FEFF as 84 31 95 33. The explorer has no GB18030 format yet, so such a file is read as a code page; the explorer notes the mark and does not apply its Chinese guess, since the text is not GBK.

Encoding

00000000  43 61 66 E9 20 93 71 75 6F 74 65 64 94 20 96 20  Caf. .quoted. . 
00000010  35 B0 0D 0A                                      5...

Lines

A line is the characters up to a line ending. The explorer counts one line for each line ending, plus one for any text after the last, so a␊b␊ and a␊b are both two lines. Lines are numbered from 1, as editors number them.

POSIX defines a line as ending in a newline, so a last line with no line ending is, strictly, incomplete. It is common and harmless, and the explorer gives it as a note, not a problem.

In a single-byte code page a character is a byte, so the counts of characters are counts of bytes, not counting the line endings; the explorer shows the number of bytes above 7F rather than of non-ASCII characters. When it guesses a double-byte code page it reads the file again in that code page, so that the counts are of characters. The longest line is measured in characters. Tabs (09) and form feeds (0C, page breaks) are counted and not reported.

Line endings

In a code page only CR and LF end lines:

The byte 85 is NEL (next line) in ISO 8859-1, but an ellipsis, …, in Windows-1252, and a letter in most other code pages, so it is not taken as a line ending. A file in Mac OS Roman usually ends its lines in CR alone, and the explorer’s guess uses that (see The guess).

A file with more than one kind of ending is reported as having mixed line endings, with the first line that differs from the commonest kind.

End of the file

The text stops at the end of the file. Whether the last line has a line ending is covered under Lines. Two other things can follow the text:

• Ctrl-Z. CP/M stored file sizes in 128-byte records, so it marked the end of the text with the byte 1A (SUB, Ctrl-Z), and MS-DOS programs kept writing it. It is common on old DOS text files, and the files examined for this site included some. The explorer shows trailing 1A bytes as an End-of-file mark. A 1A inside the text is a control character.
• Zero padding. NUL bytes after the text, left by programs that preallocate a file or write it in fixed-size blocks. The explorer shows them as Zero padding and reports them. A file that is mostly such zeros is not taken as text.

Problems

In a single-byte code page every byte is a character, so there are no invalid sequences. Some bytes are unwelcome all the same: control codes, and the few bytes a code page leaves unassigned, such as 81, 8D, 8F, 90 and 9D in Windows-1252. A reader shows them as boxes, as nothing, or as U+FFFD. In a double-byte code page a lead byte with no trail byte after it is an invalid sequence.

The worse problem is the wrong code page, which no reader can detect: every byte still reads as something. Text read in the wrong code page shows “mojibake”. é shown as È is Windows-1252 read as Mac OS Roman; é shown as é is UTF-8 read as Windows-1252.

The explorer adds notes when:

• the guess is Windows-1252 and the file holds bytes that Windows-1252 leaves unassigned, so the guess may be wrong;
• most multi-byte sequences in the first 4 MB are valid UTF-8: the file is then probably UTF-8 with a few damaged or mixed-in bytes;
• the file starts with 84 31 95 33, GB18030’s byte order mark;
• the bytes look like EBCDIC, IBM mainframe text: the space, 40, is very common, most bytes are above 80, and there are no line endings. The explorer does not decode EBCDIC.

Telling it apart

There is no positive test for a code page. This format is what is left: the explorer tries it last, after UTF-8, UTF-32 and UTF-16, and its parser accepts any file that none of those took and that still looks like text. NUL and control characters together must be under one in fifty of the first 64 KB. (Tab, LF, form feed, CR, escape and Ctrl-Z do not count.) That is tighter than the Unicode formats’ test, because a code page accepts any byte, so binary data would otherwise pass. A file that is mostly zero bytes is turned down.

Most files come here because UTF-8 turned them down: an accented letter written as one byte, such as E9 for é, is a UTF-8 lead byte with no continuation. A file of pure ASCII never comes here; it is valid UTF-8. Which code page it is, is then guessed.

Code pages

Bytes 00 to 7F are ASCII in all of these. Some of the bytes above, in the four Western code pages the explorer guesses between:

And in the three Cyrillic code pages:

Unicode’s mapping files and the WHATWG Encoding Standard give the full tables.

The guess

No rule can tell these code pages apart. Every byte is valid in each of them, so the only evidence is which reading looks more like real text. Tools such as chardet, uchardet and ICU’s detector all work this way, from statistics, and all can be wrong. The explorer’s guess is the same kind of thing. Step by step:

1. Count the bytes. Every byte value is counted over the whole file, and so is how often each byte follows itself.
2. Density. The bytes above 7F are compared with the ASCII letters. In Western European text they are a few accented letters among many ASCII ones. In Russian text nearly every letter is one, so when they make up more than 0.35 of the two together, the text is taken to be mostly non-Latin, and the guess is between Windows-1251, KOI8-R and DOS 866, with DOS 437 kept in for box drawing. Otherwise it is between Windows-1252, Mac OS Roman and DOS 437.
3. Score each candidate. Each byte above 7F is read in the candidate code page and weighted by how common that character is in ordinary text: Western European letters (é, à, ü, ñ, ø …) and English typography (’ “ ” – … •), or Russian letter frequencies for the Cyrillic code pages. A C1 control code or an unassigned byte counts −3, a symbol −0.5. A letter that follows itself is not counted, since letters seldom double.
4. Box drawing. Box-drawing characters count only when they repeat (═════), as in frames and shading; a lone one counts slightly against.
5. Line endings. Lines that end in CR, with no CR LF and hardly any LF, point to the classic Mac OS, and Mac OS Roman’s score is raised.
6. The winner. The highest score wins. If it is Windows-1252 and the file has no bytes from 80 to 9F, the result is named “Windows-1252 or ISO 8859-1”, since the two agree on every other byte. With fewer than 20 bytes above 7F the explorer says the guess is weak.
7. Box art. In a dense file, when bytes B0 to DF, box drawing in both DOS code pages, are most of the bytes above 7F and at least three in ten of them repeat the byte before, the file is DOS 437, or 866 if 866’s Cyrillic letters are common as well. A listing from the DOS tree command is the typical case.
8. East Asian code pages. Also in a dense file, the start (up to 64 KB, cut at a line feed) is decoded strictly in GBK, Big5, Shift_JIS, EUC-JP and EUC-KR. A code page counts only if the start decodes without an error and at least half of what is not ASCII comes out as kana, Chinese characters or Hangul. Among those, the one whose text holds the most hiragana (for Japanese), the most of the commonest Chinese characters (的 一 是 不 了 …), or the most of the commonest Korean syllables (이 다 는 의 에 …) wins. The file is then read again in it.

The explorer shows the result as the encoding, marked “a guess”, with its reasons, and a table of the commonest bytes above 7F with what each would be in every candidate. When the guess looks wrong, that table usually shows the right code page.

In spot checks on the files examined for this site the guess read correctly DOS readmes with box drawing and tree listings (437), Mac OS 7 era release notes (Mac OS Roman, with • and ∫ as bullets), Russian and Bulgarian documentation (1251), Russian Unix documentation (KOI8-R), Chinese readmes (GBK) and Japanese readmes (Shift_JIS). Short files with only one or two accented letters remain the hardest case.

Variants

• UTF-8 text – one to four bytes a character, ASCII unchanged
• UTF-16LE text – two or four bytes a character, low byte first
• UTF-16BE text – two or four bytes a character, high byte first
• UTF-32LE text – four bytes a character, low byte first
• UTF-32BE text – four bytes a character, high byte first
• Legacy code page text – a pre-Unicode code page that the file does not name

Links