TXT file format: UTF-EBCDIC text

Introduction

UTF-EBCDIC is Unicode Technical Report #16’s encoding of Unicode for EBCDIC systems. It is UTF-8 with two changes. First, a modified UTF-8 (UTF-8-Mod) gives an intermediate I8 sequence in which the C1 controls U+0080 to U+009F are single bytes too, so that every control, the space and all the ASCII graphic characters are one byte. Second, each I8 byte is replaced by a UTF-EBCDIC byte through a fixed map, which puts those 160 characters where EBCDIC code page 1047 has them (with LF and NL swapped, below). The report has been stabilised, no update is planned, and IBM systems usually use UTF-16 instead. No browser decodes it. A file dropped on any page of this site opens in the explorer, which does.

Structure

4Signature DD 73 66 73, U+FEFF, optional4 bytes
0Lines characters of one to five bytes, each line followed by its line endingvaries

A UTF-EBCDIC text file

Signature

U+FEFF is the I8 sequence F1 BF B7 BF (four bytes, since it is above U+3FFF), which the map turns into DD 73 66 73. The explorer takes a file as UTF-EBCDIC only if it starts with those four bytes. Without them the file reads as EBCDIC, if it looks like it.

Encoding

Step 1: the I8 sequence

Code points	I8 bytes (bit patterns)
U+0000–U+007F	`0xxxxxxx`
U+0080–U+009F	`100xxxxx`
U+00A0–U+03FF	`110yyyyy 101xxxxx`
U+0400–U+3FFF	`1110zzzz 101yyyyy 101xxxxx`
U+4000–U+3FFFF	`11110www 101zzzzz 101yyyyy 101xxxxx`
U+40000–U+10FFFF	`1111100r 101wwwww 101zzzzz 101yyyyy 101xxxxx`

Every trail byte is A0 to BF. A decoder need not reject a longer form than needed; the explorer does not.

Step 2: the byte map

UTR #16 table 2: the UTF-EBCDIC byte for each I8 byte. Read the row for the high nibble of the I8 byte and the column for its low nibble. The single bytes of 00 to 9F are code page 1047’s except that LF (I8 0A) is 15 and NL, U+0085, (I8 85) is 25, the reverse of the usual EBCDIC pairing.

I8	_0	_1	_2	_3	_4	_5	_6	_7	_8	_9	_A	_B	_C	_D	_E	_F
0_	00	01	02	03	37	2D	2E	2F	16	05	15	0B	0C	0D	0E	0F
1_	10	11	12	13	3C	3D	32	26	18	19	3F	27	1C	1D	1E	1F
2_	40	5A	7F	7B	5B	6C	50	7D	4D	5D	5C	4E	6B	60	4B	61
3_	F0	F1	F2	F3	F4	F5	F6	F7	F8	F9	7A	5E	4C	7E	6E	6F
4_	7C	C1	C2	C3	C4	C5	C6	C7	C8	C9	D1	D2	D3	D4	D5	D6
5_	D7	D8	D9	E2	E3	E4	E5	E6	E7	E8	E9	AD	E0	BD	5F	6D
6_	79	81	82	83	84	85	86	87	88	89	91	92	93	94	95	96
7_	97	98	99	A2	A3	A4	A5	A6	A7	A8	A9	C0	4F	D0	A1	07
8_	20	21	22	23	24	25	06	17	28	29	2A	2B	2C	09	0A	1B
9_	30	31	1A	33	34	35	36	08	38	39	3A	3B	04	14	3E	FF
A_	41	42	43	44	45	46	47	48	49	4A	51	52	53	54	55	56
B_	57	58	59	62	63	64	65	66	67	68	69	6A	70	71	72	73
C_	74	75	76	77	78	80	8A	8B	8C	8D	8E	8F	90	9A	9B	9C
D_	9D	9E	9F	A0	AA	AB	AC	AE	AF	B0	B1	B2	B3	B4	B5	B6
E_	B7	B8	B9	BA	BB	BC	BE	BF	CA	CB	CC	CD	CE	CF	DA	DB
F_	DC	DD	DE	DF	E1	EA	EB	EC	ED	EE	EF	FA	FB	FC	FD	FE

The map is one-to-one, so the bytes of a UTF-EBCDIC sequence are in the same order as its code points. The lead bytes of an I8 sequence are C0 to F9 and its trail bytes A0 to BF; the map sends all of those to bytes that are not used by the single-byte characters.

Lines

The explorer decodes the first 16 MB of the file at once and splits the characters into lines.

Line endings

Name	Bytes	Notes
LF	`15`	U+000A
NEL	`25`	U+0085, the EBCDIC new line
CR LF	`0D 15`
CR	`0D`

Problems

Problem	Meaning
invalid byte sequence	a sequence cut short by the end of the file; a lead byte followed by a byte that is not a trail byte; a trail byte with no lead byte; or a value above U+10FFFF
unpaired surrogate	a surrogate code point
noncharacter, control character, mixed line endings	as in other text

Telling it apart

Only by the signature DD 73 66 73; the explorer tries this variant before UTF-8 and before the EBCDIC code pages. Characters beyond the first 16 MB of a larger file are not decoded.

Variants

UTF-8 text – one to four bytes a character, ASCII unchanged
UTF-16LE text – two or four bytes a character, low byte first
UTF-16BE text – two or four bytes a character, high byte first
UTF-32LE text – four bytes a character, low byte first
UTF-32BE text – four bytes a character, high byte first
Legacy code page text – a pre-Unicode code page that the file does not name
UTF-7 text – Unicode as pure ASCII, with Base64 runs
ISO-2022 and HZ text – 7-bit Japanese, Korean and Chinese with escape sequences
GB18030 text – China’s national encoding, one, two or four bytes a character
EBCDIC text – IBM mainframe text, not based on ASCII
SCSU text – Unicode compressed with windows and tags
BOCU-1 text – Unicode as differences between code points
UTF-1 text – the withdrawn ISO 10646 transformation
UTF-EBCDIC text – UTF-8 for EBCDIC systems

.TXT File Format