TXT file format: UTF-16BE text

Introduction

UTF-16BE stores Unicode text in 16-bit code units, each written high byte first (big-endian). Most characters take one unit, two bytes; characters above U+FFFF take two units, a surrogate pair, four bytes. Big-endian is the byte order of network protocols and of many older processors, and it is the default for UTF-16: text labelled only “UTF-16”, with no byte order mark, is big-endian. Java’s UTF-16 charset, for one, writes big-endian with a byte order mark. As files, though, UTF-16BE text is rare; none of the text files examined for this site used it.

It began with Unicode 1.0 (1991), whose characters were all 16 bits wide; that fixed form is now called UCS-2. Unicode 2.0 (1996) added surrogate pairs to reach beyond U+FFFF, which made it UTF-16. It is defined in chapter 3 of the Unicode Standard and in RFC 2781.

A UTF-16BE text file has no header and no length fields: a byte order mark, usually, then code units to the end of the file. The same text in UTF-16LE has every pair of bytes the other way round, and in UTF-8 it has different bytes altogether, so each is a format of its own.

A file dropped on any page of this site opens in the explorer, which shows its structure (the byte order mark, each line, and for a line beyond ASCII each character with its bytes) and a preview of the text.

Structure

The file has two parts. Its length is a whole number of 2-byte units.

0Byte order mark FE FF, usually present0 or 2 bytes
0 or 2Lines characters of one or two 16-bit units, each line followed by its line endingvaries

The parts of a UTF-16BE text file, in order

Byte order mark

The character U+FEFF, ZERO WIDTH NO-BREAK SPACE, written at the very start of a file, is the byte order mark (BOM). High byte first, it is FE FF. Read the other way round, those bytes would be U+FFFE, which is a noncharacter that never appears in text, so the mark settles the byte order. It is not part of the text, and a reader drops it.

Without a mark, the Unicode FAQ and RFC 2781 say that text labelled only “UTF-16” is big-endian. A file has no label, though, and most UTF-16 files are little-endian from Windows, so the explorer decides by the bytes, not by the default (see Telling it apart).

First bytes	Encoding
`FE FF`	UTF-16BE (this format)
`FF FE`	UTF-16LE
`EF BB BF`	UTF-8
`00 00 FE FF`	UTF-32BE
`FF FE 00 00`	UTF-32LE

A U+FEFF further on in the text (FE FF in the middle of a file, on an even offset) is usually a byte order mark left behind where two files were joined. The explorer reports it as a problem.

00000000  FE FF 00 48 00 65                                ...H.e

“He” in UTF-16BE, after its byte order mark

Encoding

Each character is one or two 16-bit code units, and each unit is written as two bytes, high byte first:

Code points	Units	Bytes	How
U+0000 to U+D7FF, U+E000 to U+FFFF	1	2	the code point itself
U+D800 to U+DFFF	none: these values are the surrogates, kept for pairs, and are not characters
U+10000 to U+10FFFF	2	4	a high surrogate (`D800`–`DBFF`), then a low surrogate (`DC00`–`DFFF`)

Surrogate pairs

To write a character above U+FFFF:

Subtract 0x10000 from the code point. This leaves a 20-bit number.
Its high 10 bits plus 0xD800 make the high surrogate.
Its low 10 bits plus 0xDC00 make the low surrogate.
Write the high surrogate, then the low one, each high byte first.

To read one: a unit from D800 to DBFF must be followed by one from DC00 to DFFF; the code point is 0x10000 + ((high − 0xD800) × 0x400) + (low − 0xDC00).

For 😀 U+1F600: 0x1F600 − 0x10000 = 0xF600, which is 0000111101 1000000000 in two groups of ten bits. The high group is 0x3D, so the high surrogate is D83D; the low group is 0x200, so the low surrogate is DE00. High byte first, that is D8 3D DE 00.

Worked examples

Character	Code point	Units	UTF-16BE
A	U+0041	`0041`	`00 41`
é	U+00E9	`00E9`	`00 E9`
€	U+20AC	`20AC`	`20 AC`
中	U+4E2D	`4E2D`	`4E 2D`
😀	U+1F600	`D83D DE00`	`D8 3D DE 00`

This 34-byte file holds two lines, “Café €4” and “中文 😀”, each ending in CR LF. The byte order mark and the surrogate pair are marked:

00000000  FE FF 00 43 00 61 00 66 00 E9 00 20 20 AC 00 34  ...C.a.f...  ..4
00000010  00 0D 00 0A 4E 2D 65 87 00 20 D8 3D DE 00 00 0D  ....N-e.. .=....
00000020  00 0A                                            ..

“Café €4” and “中文 😀” in UTF-16BE, with a byte order mark and CR LF line endings

Properties

Text that is mostly ASCII or Latin has a zero in every first byte of a pair: the high byte of each unit. That is how such a file is recognised without a byte order mark, and why programs that treat a zero byte as the end of a string read it as empty.
Chinese, Japanese and Korean characters, nearly all between U+0800 and U+FFFF, take two bytes each, against three in UTF-8. ASCII takes two bytes against one.
A reader must start on an even offset. One byte lost or added shifts every character after it.
A high surrogate can always be told from a low one, so a reader that starts in the middle of a pair can find the next character.

Lines

A line is the characters up to a line ending. The explorer counts one line for each line ending, plus one for any text after the last, so a␊b␊ and a␊b are both two lines. Lines are numbered from 1, as editors number them.

POSIX defines a line as ending in a newline, so a last line with no line ending is, strictly, incomplete. It is common and harmless, and the explorer gives it as a note, not a problem.

Characters are counted as code points, not units or bytes, and not counting the line endings: a surrogate pair is one character, and an unpaired surrogate counts as one. The longest line is measured in characters. Tabs (00 09) and form feeds (00 0C, page breaks) are counted and not reported.

Line endings

Each line-ending character is one code unit, so each takes two bytes, high byte first. Unicode’s newline guidelines (chapter 5 of the Standard) list CR, LF, CR LF and NEL as newline functions, and LS and PS as separators; the explorer ends a line at any of the six:

Name	Code point	UTF-16BE bytes	Where it is used
LF (line feed)	U+000A	`00 0A`	Unix, Linux, macOS
CR LF	U+000D U+000A	`00 0D 00 0A`	Windows, MS-DOS, Internet protocols
CR (carriage return)	U+000D	`00 0D`	classic Mac OS (up to 9)
NEL (next line)	U+0085	`00 85`	EBCDIC systems’ NL, mapped to Unicode
LS (line separator)	U+2028	`20 28`	Unicode only, rare
PS (paragraph separator)	U+2029	`20 29`	Unicode only, rare

A file with more than one kind of ending is reported as having mixed line endings, with the first line that differs from the commonest kind. The vertical tab and the form feed do not end lines.

End of the file

The text stops at the end of the file. There is no end-of-file mark in UTF-16 files: the Ctrl-Z of MS-DOS belongs to byte-oriented text. Zero units at the end (00 00) are NUL characters, and the explorer reports them as such.

A file of odd length ends in half a code unit. The explorer reports that last byte as a character cut short by the end of the file; browsers turn it into U+FFFD.

Problems

Every pair of bytes is a code unit, so UTF-16 has no invalid byte values. What can go wrong is the pairing of surrogates:

An unpaired surrogate: a high surrogate (D800 to DBFF) not followed by a low one, or a low surrogate (DC00 to DFFF) on its own. The Unicode FAQ says unpaired surrogates are invalid in UTFs. They turn up where text was built from 16-bit units without checking, as JavaScript strings and Windows file names can be. D8 3D 00 41 is a high surrogate followed by “A”.
Half a unit at the end of a file of odd length.

A strict decoder stops with an error. Browsers, following the WHATWG Encoding Standard, turn an unpaired surrogate or an odd last byte into U+FFFD, the replacement character �. The explorer keeps the unit, shows its value and reports it.

Problem	Meaning
unpaired surrogate	half of a UTF-16 pair with no other half
NUL character	U+0000 in the text
control character	a C0 control other than tab, LF, form feed and CR; DEL; or a C1 control, U+0080 to U+009F, other than NEL
U+FEFF inside the text	a leftover byte order mark
character cut short by the end of the file	a last byte with no partner
mixed line endings	more than one kind of line ending

Telling it apart

The explorer’s parser for this format accepts a file that starts with FE FF. It turns down a file that starts with FF FE (the UTF-16LE mark) or EF BB BF (the UTF-8 mark). Without a mark it looks at the first 64 KB:

The zero bytes. In mostly ASCII or Latin text, the first byte of each pair, the high byte, is zero. At least a quarter of the even-numbered bytes (counting from 0) must be zero, and the odd-numbered bytes may have at most a twentieth as many zeros.
It starts as text. The first eight units must each be at most 00FF. Binary files that hold UTF-16 strings have the zero pattern too, but not at their start.
It is not binary. NUL in more than one character in a hundred, or other control characters in more than one in ten, turns it down.

UTF-16BE text in Chinese, Japanese or Korean without a byte order mark has few zero bytes, and cannot be found this way. The explorer goes by the zero bytes rather than the big-endian default, because a file of real text shows its byte order plainly. UTF-32 is tried before UTF-16.

Variants

UTF-8 text – one to four bytes a character, ASCII unchanged
UTF-16LE text – two or four bytes a character, low byte first
UTF-16BE text – two or four bytes a character, high byte first
UTF-32LE text – four bytes a character, low byte first
UTF-32BE text – four bytes a character, high byte first
Legacy code page text – a pre-Unicode code page that the file does not name

.TXT File Format