Everything after the header is one object in Python’s marshal format: the
module’s code object, which holds
further objects. An object is one type byte and then whatever that type needs. All
numbers are little-endian. CPython 3.3 writes marshal version 2. CPython 3.4 and 3.5
write version 4, which adds shorter forms for strings and tuples and a way to refer back
to an earlier object.
The FLAG_REF bit (3.4 and 3.5)
In a file from 3.4 or 3.5 the top bit of a type byte, 0x80, is FLAG_REF.
The low seven bits are the type. When the bit is set, the reader adds the object to a
list as it reads it, and a later r object can use
the object again by its index in that list. The list has no size in the file: it grows
by one for each flagged object, in the order they are met. A container takes its place
before its items are read, so its index is lower than theirs.
CPython sets the bit only on an object something else also refers to, so the same value can appear with or without it. A strict reader must keep the list exact, because one missed entry shifts every later index. CPython 3.3 never sets the bit.
For example, these bytes, constructed from the rules above, are a tuple holding the
string 'ab' twice, as CPython 3.4 and 3.5 write it:
| Bytes | Meaning |
|---|---|
29 02 | ) small tuple of 2 items, no flag |
DA 02 61 62 | Z (5A with the top bit) short interned ASCII, 2 bytes, ab; saved as index 0 |
72 00 00 00 00 | r: the object at index 0 |
CPython 3.3 has no references, so the same value takes
28 02 00 00 00 and then 75 02 00 00 00 61 62 twice.
Object types
“Written” says which releases write the type. “Read only” means it is accepted but not written.
| Byte | Type | After the type byte | Written | Used for |
|---|---|---|---|---|
0 | TYPE_NULL | Nothing | 3.3–3.5 | Marks the end of a dictionary |
N | TYPE_NONE | Nothing | 3.3–3.5 | None |
F | TYPE_FALSE | Nothing | 3.3–3.5 | False |
T | TYPE_TRUE | Nothing | 3.3–3.5 | True |
S | TYPE_STOPITER | Nothing | 3.3–3.5 | StopIteration |
. | TYPE_ELLIPSIS | Nothing | 3.3–3.5 | Ellipsis |
i | TYPE_INT | 4 bytes, signed | 3.3–3.5 | An integer that fits in 32 bits |
I | TYPE_INT64 | 8 bytes, signed | None | Read only, by 3.3. A 64-bit integer; the 3.4 and 3.5 sources have no such type |
l | TYPE_LONG | A signed 32-bit count of digits, then 2 bytes per digit | 3.3–3.5 | An integer of any size, base 215, lowest digit first; the count is negative for a negative number |
f | TYPE_FLOAT | A 1-byte length, then text | None | Read only. A float as text, from marshal versions 0 and 1 |
g | TYPE_BINARY_FLOAT | 8 bytes, IEEE 754 double | 3.3–3.5 | A float |
x | TYPE_COMPLEX | A 1-byte length and text, twice: real then imaginary | None | Read only. A complex number as text |
y | TYPE_BINARY_COMPLEX | Two 8-byte doubles: real, imaginary | 3.3–3.5 | A complex number |
s | TYPE_STRING | A signed 32-bit length, then the bytes | 3.3–3.5 | A bytes object, such as co_code and co_lnotab |
t | TYPE_INTERNED | A signed 32-bit length, then UTF-8 | 3.4, 3.5 | An interned string with non-ASCII characters. CPython 3.3 has no such type |
R | TYPE_STRINGREF | 4 bytes, index | None | Not a type of this format. CPython 3.3 to 3.5 neither write nor read it; it belongs to Python 2. The explorer has no strings to look up here, so such a reference fails |
u | TYPE_UNICODE | A signed 32-bit length, then UTF-8 (with lone surrogates kept) | 3.3–3.5 | Text. In 3.3, every string; in 3.4 and 3.5, a string with non-ASCII characters that is not interned |
a | TYPE_ASCII | A signed 32-bit length, then the bytes | 3.4, 3.5 | An ASCII string of 256 characters or more |
A | TYPE_ASCII_INTERNED | A signed 32-bit length, then the bytes | 3.4, 3.5 | An interned ASCII string of 256 characters or more |
z | TYPE_SHORT_ASCII | A 1-byte length, then the bytes | 3.4, 3.5 | An ASCII string of up to 255 characters |
Z | TYPE_SHORT_ASCII_INTERNED | A 1-byte length, then the bytes | 3.4, 3.5 | An interned ASCII string of up to 255 characters: most names |
( | TYPE_TUPLE | A signed 32-bit count, then the items | 3.3–3.5 | A tuple: every tuple in 3.3; in 3.4 and 3.5, one of 256 items or more |
) | TYPE_SMALL_TUPLE | A 1-byte count, then the items | 3.4, 3.5 | A tuple of up to 255 items |
[ | TYPE_LIST | A signed 32-bit count, then the items | 3.3–3.5 | A list |
{ | TYPE_DICT | Key and value objects in turn, ending with 0 | 3.3–3.5 | A dictionary |
< | TYPE_SET | A signed 32-bit count, then the items | 3.3–3.5 | A set |
> | TYPE_FROZENSET | A signed 32-bit count, then the items | 3.3–3.5 | A frozen set, as made for a constant set in code |
c | TYPE_CODE | The fields of a code object | 3.3–3.5 | A module, class, function or comprehension body |
r | TYPE_REF | 4 bytes, unsigned index | 3.4, 3.5 | Another use of a saved object; see FLAG_REF |
: | TYPE_SLICE | Three objects: start, stop, step | None | Not a type of this format. CPython 3.3 to 3.5 neither write nor read it; the explorer accepts the byte because a later release uses it |
A byte that names no type makes CPython stop with “bad marshal data (unknown type code)”. The explorer reports it too. The explorer also accepts the 3.4 and 3.5 types in a 3.3 file, which CPython 3.3 itself would refuse.