TXT file format: SCSU text

Introduction

SCSU, the Standard Compression Scheme for Unicode, is defined in Unicode Technical Standard #6. It writes Unicode text in about as many bytes as a legacy code page would: text in one small alphabet takes one byte a character, and a tag byte moves a 128-character window to the alphabet in use. It is rare as a file on a PC, and the standard has been stabilised, with no further updates planned. No browser decodes it. A file dropped on any page of this site opens in the explorer, which does.

Structure

The file is a stream of bytes that are either tags or text, with state carried through the whole file: a mode (single-byte or Unicode), eight dynamic windows and the active one. Line endings are the characters LF, CR and CR LF, written as bytes where the mode and window allow, so the lines cannot be found without decoding the tags from the start.

3Signature 0E FE FF, U+FEFF quoted, optional3 bytes
0Lines characters and tags, each line followed by its line endingvaries

An SCSU text file

Signature

An initial U+FEFF is encoded as 0E FE FF (SQU, “quote Unicode”, then the character), which UTS #6 recommends because it is the only form that does not change the decoder’s state. The explorer takes a file as SCSU only if it starts with these three bytes; SCSU has no other mark, and ordinary compressed text could be read as many other encodings. The decoder reads the other legal forms of an initial U+FEFF (0F FE FF, or 18 A5 FF to 1F A5 FF, which define a window) as the character they encode, but does not recognise a file by them.

Encoding

The decoder starts in single-byte mode with dynamic window 0 active and the windows at the positions of the table below. A window is 128 characters long.

Single-byte mode

Bytes	Tag	Meaning
`00`, `09`, `0A`, `0D`, `20`–`7F`		the character with that code (U+0000, tab, LF, CR and ASCII)
`01`–`08`	SQ0–SQ7	quote: the next byte is from window n once; `00`–`7F` from the static window n, `80`–`FF` from the dynamic window n
`0B`	SDX	two bytes follow, h and l: window h >> 5 is set to `10000` + `80` × ((h & `1F`) × `100` + l) and becomes active
`0C`		reserved
`0E`	SQU	the next two bytes are one BMP character, high byte first
`0F`	SCU	switch to Unicode mode
`10`–`17`	SC0–SC7	make dynamic window n active
`18`–`1F`	SD0–SD7	one byte follows, an index into the window offset table: define window n there and make it active
`80`–`FF`		the active dynamic window’s offset + (byte − `80`)

Unicode mode

Each character is two bytes, big-endian UTF-16, except for these tags:

Bytes	Tag	Meaning
`E0`–`E7`	UC0–UC7	back to single-byte mode, window n active
`E8`–`EF`	UD0–UD7	one index byte follows; define window n and return to single-byte mode with it active
`F0`	UQU	the next two bytes are one BMP character, even if the first would be a tag
`F1`	UDX	like SDX, then single-byte mode
`F2`		reserved
anything else		the first of two bytes of a character

A supplementary character is a surrogate pair in Unicode mode, two SQU-quoted halves in single-byte mode, or one byte in a window set with SDX or UDX. The explorer joins the halves of a pair into one character however they were written.

Windows

Window	Static offset	Dynamic offset at the start
0	`0000`	`0080` Latin-1 Supplement
1	`0080`	`00C0` Latin letters
2	`0100`	`0400` Cyrillic
3	`0300`	`0600` Arabic
4	`2000`	`0900` Devanagari
5	`2080`	`3040` Hiragana
6	`2100`	`30A0` Katakana
7	`3000`	`FF00` fullwidth ASCII

The window offset table

Index byte	Window starts at
`00`	reserved
`01`–`67`	index × `80`: U+0080 to U+3380
`68`–`A7`	index × `80` + `AC00`: U+E000 to U+FF80
`A8`–`F8`	reserved
`F9`	`00C0`
`FA`	`0250` IPA Extensions
`FB`	`0370` Greek
`FC`	`0530` Armenian
`FD`	`3040` Hiragana
`FE`	`30A0` Katakana
`FF`	`FF60` halfwidth Katakana

Examples

From UTS #6 section 9, all decoded correctly by the explorer (the test decodes them): 12 9C BE C1 BA B2 B0 is Москва (SC2 makes window 2, Cyrillic at U+0400, active; 9C is U+041C and so on); the 178 bytes of its Japanese sample are 116 characters; and its “all features” string of 35 bytes holds every tag.

Lines

The explorer decodes the whole file (the first 16 MB) once, then splits the characters into lines at LF, CR and CR LF. A line’s characters and the tags between them are listed with their bytes.

Line endings

Name	Characters
LF	U+000A
CR LF	U+000D U+000A
CR	U+000D

Problems

Problem	Meaning
invalid byte sequence	a tag cut short by the end of the file; byte `0C` or `F2`; SD or UD with a reserved index byte; and, from the explorer’s own checks, nothing else: every other byte sequence means something
unpaired surrogate	half of a surrogate pair with no other half
noncharacter, control character, mixed line endings	as in other text

Telling it apart

Only by the signature 0E FE FF at the start. Without it, an SCSU file reads as whatever else its bytes fit, usually a legacy code page. The explorer tries this variant before UTF-8.

What the explorer does not show: characters beyond the first 16 MB of a larger file; names of graphic characters other than those in the Unicode names file (the five Unicode pages describe it).

Variants

UTF-8 text – one to four bytes a character, ASCII unchanged
UTF-16LE text – two or four bytes a character, low byte first
UTF-16BE text – two or four bytes a character, high byte first
UTF-32LE text – four bytes a character, low byte first
UTF-32BE text – four bytes a character, high byte first
Legacy code page text – a pre-Unicode code page that the file does not name
UTF-7 text – Unicode as pure ASCII, with Base64 runs
ISO-2022 and HZ text – 7-bit Japanese, Korean and Chinese with escape sequences
GB18030 text – China’s national encoding, one, two or four bytes a character
EBCDIC text – IBM mainframe text, not based on ASCII
SCSU text – Unicode compressed with windows and tags
BOCU-1 text – Unicode as differences between code points
UTF-1 text – the withdrawn ISO 10646 transformation
UTF-EBCDIC text – UTF-8 for EBCDIC systems

.TXT File Format