.TXT File Format

Introduction

BOCU-1, Binary Ordered Compression for Unicode, is described in Unicode Technical Note #6. Each character is written as the difference from the one before, in one to four bytes, so text in one alphabet takes one or two bytes a character, and the byte order of the result follows the order of the code points. It keeps the C0 controls as themselves, which makes it usable for text files and MIME. It is rare as a file, and no browser decodes it. A file dropped on any page of this site opens in the explorer, which does.

Structure

Signature

An initial U+FEFF is encoded in BOCU-1 as FB EE 28. The explorer takes a file as BOCU-1 only if it starts with these three bytes. The signature is an ordinary character, so it sets the state like any other (the next character is a difference from U+FEC0); a control character, such as the line ending, resets it.

Encoding

The state is prev, a code point, 40 at the start.

The difference from prev is one to four bytes. The first byte tells how many follow and the range of the difference; each following byte is a trail byte worth 0 to 242, counting the twenty controls 01–06, 10–19 and 1C–1F as 0 to 19 and the bytes 21 to FF as 20 to 242 (byte − 0D). Other controls are not allowed as trail bytes.

The character is prev + the difference, and must be U+0000 to U+10FFFF. The new prev is the middle of the character’s block: for Hiragana (U+3040 to U+309F) 3070; for the Unihan ideographs (U+4E00 to U+9FA5) 7711 (U+4E00 + 10,513); for Hangul syllables (U+AC00 to U+D7A3) C1D1; otherwise the character with its low seven bits cleared, plus 40.

Check: U+FEFF from 40 is a difference of 65,215 = 10,513 + 225 × 243 + 27, which is FB then trail values 225 and 27, the bytes EE and 28.

Lines

The explorer decodes the whole file (the first 16 MB) once, and splits the characters into lines at LF, CR and CR LF. Every C0 control resets the state, so each line is decoded the same way alone, apart from the first after the signature.

Line endings

Problems

Telling it apart

Only by the signature FB EE 28. Without it a BOCU-1 file reads as a legacy code page. The explorer tries this variant before UTF-8. Characters beyond the first 16 MB of a larger file are not decoded.

Variants

• UTF-8 text – one to four bytes a character, ASCII unchanged
• UTF-16LE text – two or four bytes a character, low byte first
• UTF-16BE text – two or four bytes a character, high byte first
• UTF-32LE text – four bytes a character, low byte first
• UTF-32BE text – four bytes a character, high byte first
• Legacy code page text – a pre-Unicode code page that the file does not name
• UTF-7 text – Unicode as pure ASCII, with Base64 runs
• ISO-2022 and HZ text – 7-bit Japanese, Korean and Chinese with escape sequences
• GB18030 text – China’s national encoding, one, two or four bytes a character
• EBCDIC text – IBM mainframe text, not based on ASCII
• SCSU text – Unicode compressed with windows and tags
• BOCU-1 text – Unicode as differences between code points
• UTF-1 text – the withdrawn ISO 10646 transformation
• UTF-EBCDIC text – UTF-8 for EBCDIC systems

Links