TXT file format: ISO-2022 and HZ text

Introduction

These are 7-bit encodings of Japanese, Korean and Chinese for mail systems that only carry ASCII. Every byte is below 80. Escape sequences switch between character sets, so the same two bytes can be two ASCII letters or one Chinese character, depending on the last switch. ISO-2022-JP is RFC 1468, ISO-2022-KR RFC 1557 and ISO-2022-CN RFC 1922; all three follow the scheme of ISO/IEC 2022. HZ, RFC 1843, uses ~ sequences instead. As far as a byte stream goes they are valid ASCII, so without this format they would read as UTF-8 full of stray letters.

A file dropped on any page of this site opens in the explorer, which decodes them, shows every shift sequence as its own item, and decodes each character it can.

Structure

0Lines of bytes below 80, each interpreted through the current shift state, followed by a line endingvaries

An ISO-2022 or HZ text file: only lines

Byte order mark

These encodings have no byte order mark. Nothing but the escape sequences says the file is one of them.

Encoding

Each double-byte set writes a character as two bytes from 21 to 7E. The double-byte tables are the same as those of the 8-bit encodings (EUC-JP, EUC-KR, GB 2312), with the top bit of both bytes cleared.

ISO-2022-JP (RFC 1468)

The text starts in ASCII. An escape sequence selects a character set into G0 until the next one.

Escape sequence	Bytes	Selects	Width
`ESC ( B`	`1B 28 42`	ASCII	1
`ESC ( J`	`1B 28 4A`	JIS X 0201 Roman (the same as ASCII except `\` and `~`)	1
`ESC ( I`	`1B 28 49`	JIS X 0201 Katakana (not part of RFC 1468, but widely used)	1
`ESC $ @`	`1B 24 40`	JIS C 6226-1978 (JIS X 0208-1978)	2
`ESC $ B`	`1B 24 42`	JIS X 0208-1983	2
`ESC $ ( D`	`1B 24 28 44`	JIS X 0212-1990 (RFC 2237, not in RFC 1468)	2

RFC 1468 says the text must switch to ASCII or JIS X 0201 Roman before the end of each line, and must end in ASCII; the next line then starts in that set. The explorer starts each line in ASCII, so a line starting in Roman would show \ and ~ where the Roman set has ¥ and ‾. The double-byte characters decode through the browser’s EUC-JP decoder.

ISO-2022-KR (RFC 1557)

The designator ESC $ ) C (1B 24 29 43), once, at the start of a line, selects KS X 1001 (KSC 5601) into G1. Then shift out (SO, 0E) starts two-byte Hangul and hanja, and shift in (SI, 0F) returns to ASCII; SO and SI must pair within a line. Each line starts in ASCII. The double-byte characters decode through EUC-KR.

ISO-2022-CN and ISO-2022-CN-EXT (RFC 1922)

Designators select a character set for SO or for SS2 (ESC N, for the next two bytes only). Each line starts in ASCII and must carry its own designations, and ends in ASCII.

Escape sequence	Bytes	Set	Shift	The explorer
`ESC $ ) A`	`1B 24 29 41`	GB 2312-80	SO	decodes (through GBK)
`ESC $ ) G`	`1B 24 29 47`	CNS 11643 plane 1	SO	row and cell only
`ESC $ * H`	`1B 24 2A 48`	CNS 11643 plane 2	SS2	row and cell only

ISO-2022-CN-EXT adds GB 12345-90, ISO-IR-165, GB 7589-87, GB 13131-91, GB 7590-87, GB 13132-91 and CNS 11643 planes 3 to 7, with SS3 (ESC O). RFC 1922 leaves the GB sets’ final characters unassigned, “these GB sets shall only be used once these final characters are assigned”, and they never were. The browser has no table for CNS 11643 (Unicode’s Unihan database holds a public mapping), so the explorer shows those characters by plane, row and cell rather than as text.

HZ (RFC 1843)

The text starts in ASCII. ~{ switches to GB 2312, two bytes at a time, first byte 21–77, second 21–7E; ~} switches back to ASCII.

In ASCII mode	Meaning
`~{`	start GB mode
`~~`	a literal tilde
`~` then a newline	a line continuation: the newline is not part of the text
`~` then anything else	an invalid sequence

RFC 1843 has no line rule of its own; encoders that keep lines short end GB mode before each line break. The explorer starts each line in ASCII. The GB characters decode through GBK with the top bits set.

Lines

A line is the bytes up to CR, LF or CR LF, which are never a byte of a two-byte character (the bytes of those are 21 and above). Characters are counted as characters, not bytes; escape sequences and shift codes are not counted. Each shift sequence shows as its own item in the character list.

Line endings

Name	Bytes
LF	`0A`
CR LF	`0D 0A`
CR	`0D`

End of the file

The text simply stops. Text encoded correctly returns to ASCII before the end, as RFC 1468 requires; the explorer does not treat a missing switch as a problem.

Problems

Problem	Meaning
invalid byte sequence	an escape sequence the explorer does not know; a byte outside `21`–`7E` where a two-byte character should start; an HZ tilde that is not one of the sequences above; or a two-byte form with no character in the table
control character	a control other than tab, vertical tab, form feed and the line endings, and the shift codes that the encoding uses
mixed line endings	more than one kind of line ending

Telling it apart

The explorer tries this format before UTF-8, since these files are also valid ASCII. It accepts a file when every byte is above 00 and below 80, the file holds one of the escape sequences above, or a ~{ with HZ’s tilde rules holding throughout, and the two-byte characters reached that way decode, with no more than one error in every hundred two-byte characters. The escape sequence decides the encoding: ESC $ ) C is Korean, ESC $ ) A, G or ESC $ * H Chinese, and ESC $ @, ESC $ B, ESC ( J or ESC ( I Japanese. ESC ( B alone, which terminal programs emit, is not enough.

Variants

UTF-8 text – one to four bytes a character, ASCII unchanged
UTF-16LE text – two or four bytes a character, low byte first
UTF-16BE text – two or four bytes a character, high byte first
UTF-32LE text – four bytes a character, low byte first
UTF-32BE text – four bytes a character, high byte first
Legacy code page text – a pre-Unicode code page that the file does not name
UTF-7 text – Unicode as pure ASCII, with Base64 runs
ISO-2022 and HZ text – 7-bit Japanese, Korean and Chinese with escape sequences
GB18030 text – China’s national encoding, one, two or four bytes a character
EBCDIC text – IBM mainframe text, not based on ASCII

.TXT File Format