DirectX .x binary encoding: tokens and records

The binary encoding, introduced with DirectX 3.0, is a tokenised form of the text encoding. After the 16-byte header with format bin (or, once decompressed, bzip), the file is a stream of tokens. Stand-alone tokens give the structure, like the braces and keywords of the text form; record-bearing tokens are followed by data. All values are little-endian.

Tokens

Each token is a 16-bit little-endian number (a WORD).

Value	Hex	Name	Kind	Text equivalent
1	`01 00`	`TOKEN_NAME`	Record	A name
2	`02 00`	`TOKEN_STRING`	Record	A quoted string
3	`03 00`	`TOKEN_INTEGER`	Record	An integer, as in an array size
5	`05 00`	`TOKEN_GUID`	Record	`<UUID>`
6	`06 00`	`TOKEN_INTEGER_LIST`	Record	Integers of a data object
7	`07 00`	`TOKEN_FLOAT_LIST`	Record	Floats of a data object
10	`0A 00`	`TOKEN_OBRACE`	Stand-alone	`{`
11	`0B 00`	`TOKEN_CBRACE`	Stand-alone	`}`
12	`0C 00`	`TOKEN_OPAREN`	Stand-alone	`(`
13	`0D 00`	`TOKEN_CPAREN`	Stand-alone	`)`
14	`0E 00`	`TOKEN_OBRACKET`	Stand-alone	`[`
15	`0F 00`	`TOKEN_CBRACKET`	Stand-alone	`]`
16	`10 00`	`TOKEN_OANGLE`	Stand-alone	`<`
17	`11 00`	`TOKEN_CANGLE`	Stand-alone	`>`
18	`12 00`	`TOKEN_DOT`	Stand-alone	`.`
19	`13 00`	`TOKEN_COMMA`	Stand-alone	`,`
20	`14 00`	`TOKEN_SEMICOLON`	Stand-alone	`;`
31	`1F 00`	`TOKEN_TEMPLATE`	Stand-alone	`template`
40	`28 00`	`TOKEN_WORD`	Stand-alone	`WORD`
41	`29 00`	`TOKEN_DWORD`	Stand-alone	`DWORD`
42	`2A 00`	`TOKEN_FLOAT`	Stand-alone	`FLOAT`
43	`2B 00`	`TOKEN_DOUBLE`	Stand-alone	`DOUBLE`
44	`2C 00`	`TOKEN_CHAR`	Stand-alone	`CHAR`
45	`2D 00`	`TOKEN_UCHAR`	Stand-alone	`UCHAR`
46	`2E 00`	`TOKEN_SWORD`	Stand-alone	`SWORD`
47	`2F 00`	`TOKEN_SDWORD`	Stand-alone	`SDWORD`
48	`30 00`	`TOKEN_VOID`	Stand-alone	None; not used by the grammar
49	`31 00`	`TOKEN_LPSTR`	Stand-alone	`STRING`
50	`32 00`	`TOKEN_UNICODE`	Stand-alone	`UNICODE`
51	`33 00`	`TOKEN_CSTRING`	Stand-alone	`CSTRING`
52	`34 00`	`TOKEN_ARRAY`	Stand-alone	`array`

Values 4, 8, 9, 21 to 30 and 32 to 39 are not defined.

Token records

The six record-bearing tokens are followed by their data. Sizes are in bytes.

Token	Then	Size	Meaning
`TOKEN_NAME`	count, DWORD	4	Length of the name
`TOKEN_NAME`	name, bytes	count	ASCII name, no terminating zero
`TOKEN_STRING`	count, DWORD	4	Length of the string
	string, bytes	count	ASCII string, no terminating zero
	terminator	2 (see below)	`TOKEN_SEMICOLON` or `TOKEN_COMMA`, as the text form would have after the string
`TOKEN_INTEGER`	value, DWORD	4	One integer
`TOKEN_GUID`	Data1, DWORD	4	A GUID in its usual binary form: `3D82AB44-62DA-11CF-AB39-0020AF71E433` is stored `44 AB 82 3D DA 62 CF 11 AB 39 00 20 AF 71 E4 33`
	Data2, WORD	2
	Data3, WORD	2
	Data4, bytes	8
`TOKEN_INTEGER_LIST`	count, DWORD	4	Number of integers
`TOKEN_INTEGER_LIST`	list, DWORD each	4 × count	The integers
`TOKEN_FLOAT_LIST`	count, DWORD	4	Number of floats
`TOKEN_FLOAT_LIST`	list, float or double each	4 or 8 × count	The floats, 4 bytes each when the header says `0032` and 8 when it says `0064`

Microsoft’s token-record table gives the string terminator as a DWORD, but it is a token, and every other token is a WORD; readers such as Assimp read it as two bytes. Microsoft asks writers to join consecutive integer lists into one list, and consecutive float lists into one list. Every integer in a list takes four bytes, whatever its member’s declared type, so the WORD members of XSkinMeshHeader also take four bytes each.

Grammar of a template

template              : TOKEN_TEMPLATE name TOKEN_OBRACE class_id template_parts TOKEN_CBRACE
template_parts        : template_members_part TOKEN_OBRACKET template_option_info TOKEN_CBRACKET
                      | template_members_list
template_members_part : (empty) | template_members_list
template_option_info  : ellipsis | template_option_list
template_members_list : template_members | template_members_list template_members
template_members      : primitive | array | template_reference
primitive             : primitive_type optional_name TOKEN_SEMICOLON
array                 : TOKEN_ARRAY array_data_type name dimension_list TOKEN_SEMICOLON
template_reference    : name optional_name TOKEN_SEMICOLON
primitive_type        : TOKEN_WORD | TOKEN_DWORD | TOKEN_FLOAT | TOKEN_DOUBLE | TOKEN_CHAR
                      | TOKEN_UCHAR | TOKEN_SWORD | TOKEN_SDWORD | TOKEN_LPSTR
                      | TOKEN_UNICODE | TOKEN_CSTRING
array_data_type       : primitive_type | name
dimension_list        : dimension | dimension_list dimension
dimension             : TOKEN_OBRACKET dimension_size TOKEN_CBRACKET
dimension_size        : TOKEN_INTEGER | name
template_option_list  : template_option_part | template_option_list template_option_part
template_option_part  : name optional_class_id
name                  : TOKEN_NAME
optional_name         : (empty) | name
class_id              : TOKEN_GUID
optional_class_id     : (empty) | class_id
ellipsis              : TOKEN_DOT TOKEN_DOT TOKEN_DOT

Microsoft’s grammar for a binary template declaration. Its template_reference rule ends in YT_SEMICOLON in the original, a misprint for TOKEN_SEMICOLON.

Grammar of a data object

object          : identifier optional_name TOKEN_OBRACE optional_class_id data_parts_list TOKEN_CBRACE
data_parts_list : data_part | data_parts_list data_part
data_part       : data_reference | object | number_list | float_list | string_list
number_list     : TOKEN_INTEGER_LIST
float_list      : TOKEN_FLOAT_LIST
string_list     : string_list_1 list_separator
string_list_1   : string | string_list_1 list_separator string
list_separator  : comma | semicolon
string          : token_string
identifier      : name | primitive_type
data_reference  : TOKEN_OBRACE name optional_class_id TOKEN_CBRACE

Microsoft’s grammar for a binary data object.

Integer and float lists carry no commas or semicolons; those tokens appear only in string lists. A data object’s members are simply its numbers in order, in as few lists as the writer likes, so a reader must use the template to know which number belongs to which member. A reference may only stand for an optional (nested) member.

Worked example

Microsoft’s example data object, an instance named blue of a template RGB with three float members, in a file with 32-bit floats:

00000000  78 6F 66 20 30 33 30 32 62 69 6E 20 30 30 33 32  xof 0302bin 0032
00000010  01 00 03 00 00 00 52 47 42 01 00 04 00 00 00 62  ......RGB......b
00000020  6C 75 65 0A 00 07 00 03 00 00 00 00 00 00 00 00  lue.............
00000030  00 00 00 00 00 80 3F 0B 00                       ......?..

The header (marked), then one data object.

Offset	Bytes	Token	Meaning
16	`01 00` `03 00 00 00` `52 47 42`	`TOKEN_NAME`	Identifier `RGB`, the template
25	`01 00` `04 00 00 00` `62 6C 75 65`	`TOKEN_NAME`	Object name `blue`
35	`0A 00`	`TOKEN_OBRACE`	Start of the object
37	`07 00` `03 00 00 00`	`TOKEN_FLOAT_LIST`	Three floats follow
43	`00 00 00 00` `00 00 00 00` `00 00 80 3F`		r = 0.0, g = 0.0, b = 1.0
55	`0B 00`	`TOKEN_CBRACE`	End of the object

The same object in the text encoding is RGB blue { 0.0; 0.0; 1.0; }. The template itself would be declared, before the object, as TOKEN_TEMPLATE, the name, TOKEN_OBRACE, TOKEN_GUID and its 16 bytes, then for each member TOKEN_FLOAT, TOKEN_NAME and TOKEN_SEMICOLON, and finally TOKEN_CBRACE.

.X File Format – Binary encoding

Tokens

Token records

Grammar of a template

Grammar of a data object

Worked example