Everything after the header is one object in Python’s marshal format: the
module’s code object, which holds
further objects. An object is one type byte and then whatever that type needs. All
numbers are little-endian. CPython 3.8 and 3.9 write marshal version 4.
The FLAG_REF bit
The top bit of a type byte, 0x80, is FLAG_REF. The low seven bits are
the type. When the bit is set, the reader adds the object to a list as it reads it, and
a later r object can use the object again by its
index in that list. The list has no size in the file: it grows by one for each flagged
object, in the order they are met. A container takes its place before its items are
read, so its index is lower than theirs.
CPython sets the bit only on an object something else also refers to, so the same value can appear with or without it. A strict reader must keep the list exact, because one missed entry shifts every later index.
A real example from CPython 3.9, marshal.dumps((x, x)) where
x is the string 'ab':
| Bytes | Meaning |
|---|---|
29 02 | ) small tuple of 2 items, no flag |
DA 02 61 62 | Z (5A with the top bit) short interned ASCII, 2 bytes, ab; saved as index 0 |
72 00 00 00 00 | r: the object at index 0 |
Object types
“Read only” means CPython 3.8 and 3.9 accept the byte but do not write it.
| Byte | Type | After the type byte | Used for |
|---|---|---|---|
0 | TYPE_NULL | Nothing | Marks the end of a dictionary |
N | TYPE_NONE | Nothing | None |
F | TYPE_FALSE | Nothing | False |
T | TYPE_TRUE | Nothing | True |
S | TYPE_STOPITER | Nothing | StopIteration |
. | TYPE_ELLIPSIS | Nothing | Ellipsis |
i | TYPE_INT | 4 bytes, signed | An integer that fits in 32 bits |
I | TYPE_INT64 | 8 bytes, signed | Read only. A 64-bit integer; kept for files from older tools |
l | TYPE_LONG | A signed 32-bit count of digits, then 2 bytes per digit | An integer of any size, base 215, lowest digit first; the count is negative for a negative number |
f | TYPE_FLOAT | A 1-byte length, then text | Read only. A float as text, from marshal versions 0 and 1 |
g | TYPE_BINARY_FLOAT | 8 bytes, IEEE 754 double | A float |
x | TYPE_COMPLEX | A 1-byte length and text, twice: real then imaginary | Read only. A complex number as text |
y | TYPE_BINARY_COMPLEX | Two 8-byte doubles: real, imaginary | A complex number |
s | TYPE_STRING | A signed 32-bit length, then the bytes | A bytes object, such as co_code and co_lnotab |
t | TYPE_INTERNED | A signed 32-bit length, then UTF-8 | An interned string with non-ASCII characters |
R | TYPE_STRINGREF | 4 bytes, index | Not a type of this format. CPython 3.8 and 3.9 neither write nor read it; it belongs to Python 2. The explorer has no strings to look up here, so such a reference fails |
u | TYPE_UNICODE | A signed 32-bit length, then UTF-8 (with lone surrogates kept) | A string with non-ASCII characters that is not interned |
a | TYPE_ASCII | A signed 32-bit length, then the bytes | An ASCII string of 256 characters or more |
A | TYPE_ASCII_INTERNED | A signed 32-bit length, then the bytes | An interned ASCII string of 256 characters or more |
z | TYPE_SHORT_ASCII | A 1-byte length, then the bytes | An ASCII string of up to 255 characters |
Z | TYPE_SHORT_ASCII_INTERNED | A 1-byte length, then the bytes | An interned ASCII string of up to 255 characters: most names |
( | TYPE_TUPLE | A signed 32-bit count, then the items | A tuple of 256 items or more |
) | TYPE_SMALL_TUPLE | A 1-byte count, then the items | A tuple of up to 255 items |
[ | TYPE_LIST | A signed 32-bit count, then the items | A list |
{ | TYPE_DICT | Key and value objects in turn, ending with 0 | A dictionary |
< | TYPE_SET | A signed 32-bit count, then the items | A set |
> | TYPE_FROZENSET | A signed 32-bit count, then the items | A frozen set, as made for a constant set in code |
c | TYPE_CODE | The fields of a code object | A module, class, function or comprehension body |
r | TYPE_REF | 4 bytes, unsigned index | Another use of a saved object; see FLAG_REF |
: | TYPE_SLICE | Three objects: start, stop, step | Not a type of this format. CPython 3.8 and 3.9 neither write nor read it; the explorer accepts the byte because a later release uses it |
A byte that names no type makes CPython stop with “bad marshal data (unknown type code)”. The explorer reports it the same way.
Two more real examples from CPython 3.9. The integer 240 is
EC 03 00 00 00 00 00 00 00 00 04: type l with the top bit
set, then three digits, of which the third is 0x400. The bytes object
b'hi' is F3 02 00 00 00 68 69.