TXT file format: UTF-16LE text

Introduction

UTF-16LE stores Unicode text in 16-bit code units, each written low byte first (little-endian). Most characters take one unit, two bytes; characters above U+FFFF take two units, a surrogate pair, four bytes. It is the encoding of Windows’ own strings, and the one Windows programs mean by “Unicode”: Notepad saved it under that name for many years, and now calls it “UTF-16 LE”.

It began with Unicode 1.0 (1991), whose characters were all 16 bits wide; that fixed form is now called UCS-2. Unicode 2.0 (1996) added surrogate pairs to reach beyond U+FFFF, which made it UTF-16. It is defined in chapter 3 of the Unicode Standard and in RFC 2781.

A UTF-16LE text file has no header and no length fields: a byte order mark, usually, then code units to the end of the file. The same text in UTF-16BE has every pair of bytes the other way round, and in UTF-8 it has different bytes altogether, so each is a format of its own.

A file dropped on any page of this site opens in the explorer, which shows its structure (the byte order mark, each line, and for a line beyond ASCII each character with its bytes) and a preview of the text.

Structure

The file has two parts. Its length is a whole number of 2-byte units.

0Byte order mark FF FE, nearly always present0 or 2 bytes
0 or 2Lines characters of one or two 16-bit units, each line followed by its line endingvaries

The parts of a UTF-16LE text file, in order

Byte order mark

The character U+FEFF, ZERO WIDTH NO-BREAK SPACE, written at the very start of a file, is the byte order mark (BOM). Low byte first, it is FF FE. Read the other way round, those bytes would be U+FFFE, which is a noncharacter that never appears in text, so the mark settles the byte order. It is not part of the text, and a reader drops it.

Without a mark, the Unicode FAQ and RFC 2781 say that text labelled only “UTF-16” is big-endian. A file has no label, though, and Windows writes little-endian files with the mark; all the UTF-16LE files examined for this site had one.

FF FE 00 00 is ambiguous: it is UTF-32LE’s byte order mark, and also UTF-16LE’s followed by U+0000. The explorer tries UTF-32LE first and turns it down when what follows is not UTF-32, so such a file comes here.

First bytes	Encoding
`FF FE`	UTF-16LE (this format)
`FE FF`	UTF-16BE
`EF BB BF`	UTF-8
`FF FE 00 00`	UTF-32LE, or UTF-16LE then U+0000
`00 00 FE FF`	UTF-32BE

A U+FEFF further on in the text (FF FE in the middle of a file, on an even offset) is usually a byte order mark left behind where two files were joined. The explorer reports it as a problem.

00000000  FF FE 48 00 65 00                                ..H.e.

“He” in UTF-16LE, after its byte order mark

Encoding

Each character is one or two 16-bit code units, and each unit is written as two bytes, low byte first:

Code points	Units	Bytes	How
U+0000 to U+D7FF, U+E000 to U+FFFF	1	2	the code point itself
U+D800 to U+DFFF	none: these values are the surrogates, kept for pairs, and are not characters
U+10000 to U+10FFFF	2	4	a high surrogate (`D800`–`DBFF`), then a low surrogate (`DC00`–`DFFF`)

Surrogate pairs

To write a character above U+FFFF:

Subtract 0x10000 from the code point. This leaves a 20-bit number.
Its high 10 bits plus 0xD800 make the high surrogate.
Its low 10 bits plus 0xDC00 make the low surrogate.
Write the high surrogate, then the low one, each low byte first.

To read one: a unit from D800 to DBFF must be followed by one from DC00 to DFFF; the code point is 0x10000 + ((high − 0xD800) × 0x400) + (low − 0xDC00).

For 😀 U+1F600: 0x1F600 − 0x10000 = 0xF600, which is 0000111101 1000000000 in two groups of ten bits. The high group is 0x3D, so the high surrogate is D83D; the low group is 0x200, so the low surrogate is DE00. Low byte first, that is 3D D8 00 DE.

Worked examples

Character	Code point	Units	UTF-16LE
A	U+0041	`0041`	`41 00`
é	U+00E9	`00E9`	`E9 00`
€	U+20AC	`20AC`	`AC 20`
中	U+4E2D	`4E2D`	`2D 4E`
😀	U+1F600	`D83D DE00`	`3D D8 00 DE`

This 34-byte file, as Windows writes it, holds two lines, “Café €4” and “中文 😀”, each ending in CR LF. The byte order mark and the surrogate pair are marked:

00000000  FF FE 43 00 61 00 66 00 E9 00 20 00 AC 20 34 00  ..C.a.f... .. 4.
00000010  0D 00 0A 00 2D 4E 87 65 20 00 3D D8 00 DE 0D 00  ....-N.e .=.....
00000020  0A 00                                            ..

“Café €4” and “中文 😀” in UTF-16LE, with a byte order mark and CR LF line endings

Properties

Text that is mostly ASCII or Latin has a zero in every second byte: the high byte of each unit. That is how such a file is recognised without a byte order mark, and why programs that treat a zero byte as the end of a string cut it short.
Chinese, Japanese and Korean characters, nearly all between U+0800 and U+FFFF, take two bytes each, against three in UTF-8. ASCII takes two bytes against one.
A reader must start on an even offset. One byte lost or added shifts every character after it.
A high surrogate can always be told from a low one, so a reader that starts in the middle of a pair can find the next character.

Lines

A line is the characters up to a line ending. The explorer counts one line for each line ending, plus one for any text after the last, so a␊b␊ and a␊b are both two lines. Lines are numbered from 1, as editors number them.

POSIX defines a line as ending in a newline, so a last line with no line ending is, strictly, incomplete. It is common and harmless, and the explorer gives it as a note, not a problem.

Characters are counted as code points, not units or bytes, and not counting the line endings: a surrogate pair is one character, and an unpaired surrogate counts as one. The longest line is measured in characters. Tabs (09 00) and form feeds (0C 00, page breaks) are counted and not reported.

Line endings

Each line-ending character is one code unit, so each takes two bytes, low byte first. Unicode’s newline guidelines (chapter 5 of the Standard) list CR, LF, CR LF and NEL as newline functions, and LS and PS as separators; the explorer ends a line at any of the six:

Name	Code point	UTF-16LE bytes	Where it is used
LF (line feed)	U+000A	`0A 00`	Unix, Linux, macOS
CR LF	U+000D U+000A	`0D 00 0A 00`	Windows, MS-DOS, Internet protocols
CR (carriage return)	U+000D	`0D 00`	classic Mac OS (up to 9)
NEL (next line)	U+0085	`85 00`	EBCDIC systems’ NL, mapped to Unicode
LS (line separator)	U+2028	`28 20`	Unicode only, rare
PS (paragraph separator)	U+2029	`29 20`	Unicode only, rare

UTF-16LE files nearly all come from Windows, and most of those examined for this site used CR LF. A file with more than one kind of ending is reported as having mixed line endings, with the first line that differs from the commonest kind. The vertical tab and the form feed do not end lines.

End of the file

The text stops at the end of the file. There is no end-of-file mark in UTF-16 files: the Ctrl-Z of MS-DOS belongs to byte-oriented text. Zero units at the end (00 00) are NUL characters, and the explorer reports them as such.

A file of odd length ends in half a code unit. The explorer reports that last byte as a character cut short by the end of the file; browsers turn it into U+FFFD.

Problems

Every pair of bytes is a code unit, so UTF-16 has no invalid byte values. What can go wrong is the pairing of surrogates:

An unpaired surrogate: a high surrogate (D800 to DBFF) not followed by a low one, or a low surrogate (DC00 to DFFF) on its own. The Unicode FAQ says unpaired surrogates are invalid in UTFs. They do turn up, because Windows file names and JavaScript strings are sequences of 16-bit units that can hold them. 3D D8 41 00 is a high surrogate followed by “A”.
Half a unit at the end of a file of odd length.

A strict decoder stops with an error. Browsers, following the WHATWG Encoding Standard, turn an unpaired surrogate or an odd last byte into U+FFFD, the replacement character �. The explorer keeps the unit, shows its value and reports it.

Problem	Meaning
unpaired surrogate	half of a UTF-16 pair with no other half
NUL character	U+0000 in the text
control character	a C0 control other than tab, LF, form feed and CR; DEL; or a C1 control, U+0080 to U+009F, other than NEL
U+FEFF inside the text	a leftover byte order mark
character cut short by the end of the file	a last byte with no partner
mixed line endings	more than one kind of line ending

Telling it apart

The explorer’s parser for this format accepts a file that starts with FF FE. It turns down a file that starts with FE FF (the UTF-16BE mark) or EF BB BF (the UTF-8 mark). Without a mark it looks at the first 64 KB:

The zero bytes. In mostly ASCII or Latin text, the second byte of each pair, the high byte, is zero. At least a quarter of the odd-numbered bytes (counting from 0) must be zero, and the even-numbered bytes may have at most a twentieth as many zeros.
It starts as text. The first eight units must each be at most 00FF. Binary files that hold UTF-16 strings have the zero pattern too, but not at their start; among the files examined for this site, browser cache index files looked like UTF-16 until this check was added.
It is not binary. NUL in more than one character in a hundred, or other control characters in more than one in ten, turns it down.

UTF-16LE text in Chinese, Japanese or Korean without a byte order mark has few zero bytes, and cannot be found this way; such a file is taken for something else. In practice Windows writes the mark. UTF-32 is tried before UTF-16, because of the ambiguous FF FE 00 00.

UTF-16LE read as a single-byte code page shows a space or a dot between every letter, which is the zero byte.

Variants

UTF-8 text – one to four bytes a character, ASCII unchanged
UTF-16LE text – two or four bytes a character, low byte first
UTF-16BE text – two or four bytes a character, high byte first
UTF-32LE text – four bytes a character, low byte first
UTF-32BE text – four bytes a character, high byte first
Legacy code page text – a pre-Unicode code page that the file does not name

.TXT File Format