TXT file format: UTF-7 text

Introduction

UTF-7 writes Unicode using only 7-bit ASCII characters, so that it can travel over mail systems that strip the eighth bit. It was defined in RFC 2152 (1997), which replaced RFC 1642 (1994). It is an informational RFC, and the RFC itself says UTF-7 should normally be used only over 7-bit transports such as mail. Letters, digits and some punctuation stand for themselves; anything else is written as a run of modified Base64 holding the text’s UTF-16 code units, between a + and, usually, a -.

Browsers do not support it: the WHATWG Encoding Standard’s list of encodings is closed, and user agents must not support any other, and the usual explanation is safety, since the same bytes that one program reads as harmless ASCII can be markup to another. A UTF-7 file is pure ASCII, so without this format it would read as UTF-8 with its Base64 runs shown as text.

A file dropped on any page of this site opens in the explorer, which decodes the runs and shows each character with the bytes it came from.

Structure

The file is lines of ASCII characters. There is no header; the first bytes may be the byte order mark as a run of its own, which is the first character of the file’s first run.

0Lines direct ASCII characters and shifted runs, each line followed by its line endingvaries

A UTF-7 text file: only lines

Byte order mark

A UTF-7 file may start with U+FEFF. Written as UTF-16 in a shifted run, it starts +/v and a fourth character that depends on the two bits of the next unit, if any: +/v8, +/v9, +/v+ or +/v/. The mark carries no information about byte order, since UTF-7’s units are always written most significant byte first; it only says the text is UTF-7. In the explorer it is the first character of the first shifted run, and a note says so.

Encoding

The three sets of characters

Set	Characters	Meaning
D (direct)	`A`–`Z` `a`–`z` `0`–`9` `' ( ) , - . / : ?`	written as themselves
O (optional direct)	! " # $ % & * ; < = > @ [ ] ^ _ ` { \| }	may be written as themselves, or in a shifted run
B (modified Base64)	`A`–`Z` `a`–`z` `0`–`9` `+ /`	the characters of a shifted run; Base64 without the pad character `=`

Space, tab, CR and LF are written as themselves (rule 3). \ and ~ are left out of Set O because they are often redefined in variants of ASCII, so they must go in a shifted run; any other character must go in a run too, and so must any character beyond ASCII.

The rules

Rule 1. A character of Set D may be written directly, and a character of Set O may be too.
Rule 2. Any Unicode character may be written in a shifted run: a +, then the text’s UTF-16 code units, most significant byte first, as Set B characters, each carrying six bits. The run ends at the first character not in Set B. A - ending a run is absorbed, not part of the text; any other ending character is. Because a run also ends at a character not in Set B, a - must be written when the next character could be read as part of the run. +- is a plain +.
Rule 3. Space, tab, CR and LF are written directly.

Sixteen bits do not divide evenly by six. When a run ends, zero bits pad to a Base64 character boundary. RFC 2152 says a sequence is ill-formed if those discarded bits are not zero, and the explorer reports such a run.

A character above U+FFFF takes two code units, a surrogate pair, in the run.

Worked examples

UTF-7	Text	Notes
`A+ImIDkQ.`	A≢Α.	the run is U+2262 U+0391; it ends at the period, which is not in Set B
`Hi Mom -+Jjo--!`	Hi Mom -☺-!	the run is U+263A; `-` ends the run and is absorbed, the next `-` is the character
`+ZeVnLIqe-`	日本語	three units, U+65E5 U+672C U+8A9E
`caf+AOk-`	café	U+00E9 in a run

Lines

A line is the characters up to a line ending. Characters are counted as code points, so a run decodes to as many characters as it holds code points. The shift characters + and an absorbed - are part of a run’s bytes; in the explorer each is shown with the character it opens or closes.

Line endings

CR, LF and CR LF are written directly (rule 3), and a run ends before them, since they are not in Set B. RFC 2152 says a shifted sequence always ends at the end of a line.

Name	Bytes
LF	`0A`
CR LF	`0D 0A`
CR	`0D`

End of the file

The text simply stops at the end of the file. Ctrl-Z (1A) bytes at the very end show as an End-of-file mark and NUL bytes as Zero padding, as in other text. Anywhere else 1A is in none of the sets, and the file is not taken as UTF-7.

Problems

Problem	Meaning
invalid byte sequence	a `+` with no whole UTF-16 unit after it, or a run whose padding bits are not zero
unpaired surrogate	a run that holds half of a surrogate pair
control character	a control other than tab, LF, form feed and CR; direct characters cannot be controls, so this comes from a shifted run
noncharacter	U+FFFE, U+FFFF or one of the other 64 noncharacters in a run: valid, but not meant for interchange
U+FEFF inside the text	after the start of the first run, a zero width no-break space

Telling it apart

Nothing marks a file as UTF-7. The explorer tries this format before UTF-8, since every UTF-7 file is valid ASCII. It accepts a file only when all of this holds, in the first 64 KB:

Every byte is one of the sets above, or space, tab, CR or LF, and none is \ or ~.
Every + starts a well-formed run: at least one whole UTF-16 unit, zero padding bits, no leftover Base64 character.
At least two runs’ worth of characters beyond ASCII are present, four if they are all ideographs, unless the file starts with the byte order mark. A file that only uses + as a plus sign is ASCII, not UTF-7.
The decoded characters look like text: no control codes, unassigned code points or noncharacters. Base64 data decodes to those.

A UTF-7 file that holds only ASCII in its runs does not pass, and reads as the ASCII it is. An ASCII file that happens to pass, such as one whose only plus sign starts +AOk-, would be read as UTF-7, but that is rare. A file that starts with the byte order mark but does not pass all the checks goes to UTF-8, with a note.

Variants

UTF-8 text – one to four bytes a character, ASCII unchanged
UTF-16LE text – two or four bytes a character, low byte first
UTF-16BE text – two or four bytes a character, high byte first
UTF-32LE text – four bytes a character, low byte first
UTF-32BE text – four bytes a character, high byte first
Legacy code page text – a pre-Unicode code page that the file does not name
UTF-7 text – Unicode as pure ASCII, with Base64 runs
ISO-2022 and HZ text – 7-bit Japanese, Korean and Chinese with escape sequences
GB18030 text – China’s national encoding, one, two or four bytes a character
EBCDIC text – IBM mainframe text, not based on ASCII

.TXT File Format