.PB File Format – Length-delimited and nested messages

The LEN value

A length that runs past the end of the file, or past the end of the message that holds it, is an error. The explorer reports the field and how many bytes were left.

Strings

A string field holds UTF-8 text, with no terminator and no byte-order mark. The documentation says a string “must always contain UTF-8 encoded or 7-bit ASCII text”; proto3 parsers reject one that is not valid UTF-8. Field 8 holding the text x is 42 01 78.

Bytes

A bytes field holds any bytes. Schemas use it for binary data, such as the raw values of a tensor, and sometimes for text whose encoding the schema does not promise.

Nested messages

A field whose type is a message holds that message’s own encoding: its fields, as in a file. Nesting has no depth marker; each level ends where its length says. The documentation’s example is 1A 03 08 96 01: field 3, LEN, 3 bytes, and inside them field 1 holding 150.

An empty nested message (all its fields unset) is a LEN of 0. Because a message ends only where its length ends, a writer must know the size of each nested message before it writes it.

The 2 GiB limit

The documentation says: “Protos must be smaller than 2 GiB when serialized.” Many readers refuse anything larger. That is why large ONNX models keep their weights in external files beside the model.

Reading LEN without the schema

Without the schema the explorer has to guess what each LEN value is. It tries these in turn, and each value’s note gives the guess and the reason:

1. Empty. A length of 0 is shown as empty: an empty string, empty bytes or a message with no fields set.
2. Could it be a message? The explorer counts how many fields the bytes read as: valid tags (field number 1 or more, wire type 0, 1, 2, 3 or 5, groups matched) and values that fit, ending exactly at the end of the value.
3. Is it text? The bytes are printable UTF-8: no control characters other than tab, line feed and carriage return. For values over 64 KB, the first 64 KB are checked.
4. Text wins. When the bytes are both text and fields, they are read as text. A message’s tags and lengths are nearly always bytes below 0x20, which printable text has none of, while text that happens to read as fields is common: hi (68 69) is field 13 holding the varint 105.
5. Message, if the bytes read as one or more fields.
6. Packed varints, if the bytes read as whole varints to the end. This is a weak guess: any bytes whose last byte is below 0x80 do.
7. Bytes otherwise. If the length is a multiple of 8 or 4, the note adds that the bytes may be packed fixed-width numbers.

A settings file that comes with macOS shows how this goes. It begins 12 50 0A 47 0A 18 36 37 61 65 …:

Each guess may be wrong. A bytes field that happens to read as fields is shown as a message, and a short message of printable bytes is shown as text. A schema format, such as an ONNX model, gives every value its declared type instead.