TXT file format: UTF-1 text

Introduction

UTF-1 was an early way to turn ISO/IEC 10646 (Unicode) into a byte stream, registered as ISO-IR 178 in 1993. It did not gain acceptance and was quickly replaced by UTF-8. It keeps U+0000 to U+009F as single bytes, and writes other characters in two, three or five bytes using arithmetic modulo 190 on the trail bytes, which makes it slower to decode than UTF-8 and means ASCII punctuation can appear inside a character. Files in it are very rare. No browser decodes it. A file dropped on any page of this site opens in the explorer, which does.

Structure

3Signature F7 64 4C, U+FEFF, optional3 bytes
0Lines characters of one, two, three or five bytes, each line followed by its line endingvaries

A UTF-1 text file

Signature

U+FEFF is F7 64 4C in UTF-1. The explorer takes a file as UTF-1 only if it starts with these three bytes: UTF-1 text looks like a legacy code page otherwise.

Encoding

First byte	Bytes	Code points	Trail bytes
`00`–`9F`	1	U+0000–U+009F
`A0`	2	U+00A0–U+00FF	`A0`–`FF`: the code point itself
`A1`–`F5`	2	U+0100–U+4015	`21`–`7E`, `A0`–`FF`
`F6`–`FB`	3	U+4016–U+38E2D	two of those
`FC`–`FF`	5	U+38E2E and up	four of those

A trail byte z has the value T(z) = z − 21 for 21 to 7E (0 to 93) and z − 42 for A0 to FF (94 to 189): 190 values. Then:

two bytes (A1 to F5): U+0100 + (lead − A1) × 190 + T(trail);
three bytes: U+4016 + ((lead − F6) × 190 + T(second)) × 190 + T(third);
five bytes: U+38E2E + the same sum over four trail bytes, starting from lead − FC.

Check: U+FEFF is U+4016 + 48,873, which is lead F6 + 1 = F7, then 12,773 = 67 × 190 + 43, the trail bytes 64 and 4C. The explorer reads a value above U+10FFFF as invalid.

Lines

A line is the characters up to LF, CR or CR LF. The explorer decodes the first 16 MB of the file at once and splits it into lines.

Line endings

Name	Bytes
LF	`0A`
CR LF	`0D 0A`
CR	`0D`

Problems

Problem	Meaning
invalid byte sequence	a sequence cut short by the end of the file; a lead byte followed by a byte that is not a trail byte; `A0` followed by a byte below `A0`; or a value above U+10FFFF
unpaired surrogate	a surrogate code point
noncharacter, control character, mixed line endings	as in other text

Telling it apart

Only by the signature F7 64 4C; the explorer tries this variant before UTF-8. Characters beyond the first 16 MB of a larger file are not decoded.

Variants

UTF-8 text – one to four bytes a character, ASCII unchanged
UTF-16LE text – two or four bytes a character, low byte first
UTF-16BE text – two or four bytes a character, high byte first
UTF-32LE text – four bytes a character, low byte first
UTF-32BE text – four bytes a character, high byte first
Legacy code page text – a pre-Unicode code page that the file does not name
UTF-7 text – Unicode as pure ASCII, with Base64 runs
ISO-2022 and HZ text – 7-bit Japanese, Korean and Chinese with escape sequences
GB18030 text – China’s national encoding, one, two or four bytes a character
EBCDIC text – IBM mainframe text, not based on ASCII
SCSU text – Unicode compressed with windows and tags
BOCU-1 text – Unicode as differences between code points
UTF-1 text – the withdrawn ISO 10646 transformation
UTF-EBCDIC text – UTF-8 for EBCDIC systems

.TXT File Format