.PB File Format – Extensions and MessageSet

Extension ranges

The container reserves a subset of its field numbers for extensions:

message Foo {
  extensions 100 to 199;
  extensions 1000 to max;
}

extend Foo {
  optional int32 bar = 126;
}

A proto2 field number in an extension range is otherwise unused. Extensions cannot be maps or members of a oneof.

Choosing a number without colliding with another file’s extension is the problem extension declarations solve: the container declares which number each extension uses, with its full name, type and whether it is repeated, in the declaration list of the range’s options.

On the wire

An extension field is encoded exactly as a normal field of the same number, type and cardinality. A reader that does not know the extension sees an unknown field and keeps it. So a message with extension 126 set is a run of fields like any other, and a file cannot be told to use extensions by its bytes alone.

The explorer’s schema formats accept fields in a declared extension range even though their schema does not list them. For descriptor sets this is how custom options appear: option messages reserve 1000 to max, and an unrecognised field there is shown with its number and a note that it is in the extension range.

Custom options

Options such as [deprecated = true] are fields of the FileOptions, MessageOptions and similar messages in descriptor.proto. A custom option is an extension of one of those messages, which is why every options message ends in extension ranges. A descriptor set records custom options like any other field of the options message.

The MessageSet wire format

A message with the option message_set_wire_format = true uses a different encoding meant to match an old format used inside Google. It is deprecated, and the Protocol Buffers documentation says people outside Google will almost certainly never need it. A MessageSet must be declared extensions 4 to max;, and for it the largest extension number is 2,147,483,646.

Instead of writing an extension as an ordinary field, each one is written as a group:

The extension’s own field number appears only as the value of type_id, not as a tag. A parser that does not know it is reading a MessageSet sees a group for field 1 holding a varint and bytes, which is exactly what the explorer shows without a schema: the group’s two fields, by number.

The layout above is from the Protocol Buffers source (C++ runtime and descriptor.proto); no public page spells it out field by field, and the documentation’s language guide says only that the format exists.

Extensions and presence

Extensions have explicit presence: like a proto2 optional field, a set extension is written even if it holds its default. An unset extension is simply not written. See field presence.