.TXT File Format

Introduction

GB18030 is China’s national standard for text. It contains GB 2312 and GBK, and it covers every Unicode code point, so it is in effect a Unicode encoding. A character takes one, two or four bytes. Its editions are GB 18030-2000 (17 March 2000), GB 18030-2005 (8 November 2005, whose mandatory part has been required of software sold in China since 1 May 2006) and GB 18030-2022 (in force from 1 August 2023). The WHATWG Encoding Standard defines it for browsers, with the 2022 mapping.

A file with only one- and two-byte characters cannot be told from GBK, and is read there. A file with the byte order mark below, or with four-byte sequences, is read here.

A file dropped on any page of this site opens in the explorer, which shows its structure and decodes its characters.

Structure

Byte order mark

U+FEFF is four bytes in GB18030: 84 31 95 33. It is optional and uncommon, and like UTF-8’s mark it only says what the text is, not its byte order, because the bytes come one at a time. The explorer treats those first four bytes as the mark. A U+FEFF further on is an ordinary character.

Encoding

The second byte tells a two-byte character from a four-byte one: 30–39 starts a four-byte form, and every other valid value a two-byte form. So every byte above 7F starts a character, or continues one, by the rules above; a FF starts nothing.

pointer = (b1 − 0x81) × 12600 + (b2 − 0x30) × 1260 + (b3 − 0x81) × 10 + (b4 − 0x30)

Lines

A line is the characters up to a line ending. Characters are counted as characters, not bytes. Tabs, form feeds and vertical tabs are counted, as in ASCII text.

Line endings

Line endings are the ASCII bytes:

These bytes never occur inside a two-byte or four-byte character, since the trail bytes of two-byte forms start at 40 and four-byte forms use 30 to 39 and 81 to FE.

End of the file

Ctrl-Z (1A) at the end is shown as an End-of-file mark, and NUL bytes after the text as Zero padding, as in other byte-oriented text.

Problems

Telling it apart

The explorer reads a file as GB18030 when it starts with 84 31 95 33, or when, without a mark, the first 64 KB are valid GB18030 and hold at least one four-byte sequence. Without one a file cannot be told from GBK, and it is left to the legacy format. UTF-8 is tried first, and takes any file whose sequences are valid UTF-8.

Variants

• UTF-8 text – one to four bytes a character, ASCII unchanged
• UTF-16LE text – two or four bytes a character, low byte first
• UTF-16BE text – two or four bytes a character, high byte first
• UTF-32LE text – four bytes a character, low byte first
• UTF-32BE text – four bytes a character, high byte first
• Legacy code page text – a pre-Unicode code page that the file does not name
• UTF-7 text – Unicode as pure ASCII, with Base64 runs
• ISO-2022 and HZ text – 7-bit Japanese, Korean and Chinese with escape sequences
• GB18030 text – China’s national encoding, one, two or four bytes a character
• EBCDIC text – IBM mainframe text, not based on ASCII

Links