.ROOT File Format – RNTuple

Version number

Epoch.Major.Minor.Patch. A new epoch means backward-incompatible changes (the pre-release is epoch 0, the first public release epoch 1, and no further epoch is foreseen). A new major version means forward-incompatible changes, which must be flagged in the feature flags (for epoch 0 the major version is the release candidate number). A new minor version adds optional features that old readers ignore. A patch version is a clarification or a backport. Except for the epoch the number is for reporting; readers decide by the feature flags.

Embedding in a ROOT file and the anchor

An RNTuple's pages and envelopes are stored in invisible, non-indexed RBlob keys. No mapping from objects to keys is defined (one key may hold one page or several pages of a cluster); the only way to find anything is a locator (offset and size). The ROOT::RNTuple object is the anchor, serialised as an ordinary key. Its fields, all big-endian:

Max Key Size is the largest RBlob. A bigger payload is written as several RBlobs/TKeys and the offsets of all but the first are written at the end of the first, which gets round the 1 GiB TKey limit. The anchor may only grow by appending fields.

Compression block

Every envelope and page is wrapped in a compression block, and its compressed and uncompressed sizes must be known. If the two are equal the data is stored as is. Otherwise it is a series of chunks, each with the 9-byte header already described on the header and keys page: three algorithm bytes, a 3-byte little-endian compressed size and a 3-byte little-endian uncompressed size (at most 16,777,215, so larger data needs more chunks).

Basic types

• Integer: two's complement, little-endian, signed or unsigned, up to 64 bits.
• String: a 32-bit unsigned byte count, then UTF-8 characters.
• Compression settings: a 32-bit integer, algorithm × 100 + level, level 1 to 9 (mapped onto the algorithm's own range).

Feature flags

64-bit integers; each bit is a forward-incompatible feature in use. The top bit means more flags follow: keep reading while the signed value is negative. A reader that meets an unknown bit must stop gracefully. Defined so far:

Frames

A frame starts with a signed 64-bit size: the absolute value is the size in bytes of the whole frame including this size and the payload, and the sign says the type, negative for a list frame and positive for a record frame. A list frame then has a 32-bit number of items. Readers must use the size to skip to what follows, not add up the parts they know, so frames can be extended later.

Locators and envelope links

A locator names a byte range. The standard on-disk locator is a 32-bit size (the compressed size of the block) and a 64-bit offset from the start of the file. The size's sign bit is the type T: if it is set, the locator is a non-standard one and is read like a frame, the size being the size of the locator itself (bits 0 to 15). Then the layout is: 16-bit size, 8-bit reserved (free for the storage backend), 7-bit type (the absolute value of the 8-bit value), the T bit, then a payload. The only type so far is 0x01, the large locator, whose payload is a 64-bit content size and a 64-bit content offset. Types 0x02 to 0x7F are reserved. An envelope link is the 64-bit uncompressed size of an envelope followed by a locator.

Envelopes

An envelope is a block of metadata: a 64-bit integer whose low 16 bits are the type and whose high 48 bits are the uncompressed length, the payload, and a 64-bit XXH3 checksum over the envelope header and payload together. Types: 0x00 reserved, 0x01 header (the schema: fields and columns), 0x02 footer (clusters), 0x03 page list (locations of data pages). The compressed size comes from the anchor or the locator.

Header envelope

Contents in order: feature flags; strings for the name, the description and the identifier of the writing library or program; then four list frames, together the schema description: field record frames, column record frames, alias column record frames and extra type information frames.

Field description

A record frame of 32-bit field version, 32-bit type version, 32-bit parent field ID, 16-bit structural role and 16-bit flags; then the strings field name, type name, type alias and field description; then, according to the flags, the optional values below. A field's ID is its zero-based index in the list; subfields have larger IDs than their parent, and a top-level field names itself as parent. Field and type versions serve schema evolution.

Projection rules: source and target must have the same structural role (except an RNTupleCardinality field, whose source must be a collection); for streamer fields and plain fields the type names must be identical; variants and fixed-size arrays cannot be projected; a projected field must lie on the same path of collection fields as its source (you can project a vector of structs onto vectors of floats but not a vector of vectors of floats onto a vector of floats).

Column description

A record frame of 16-bit type, 16-bit bits on storage, 32-bit field ID, 16-bit flags and a 16-bit representation index; then, if flag 0x01 is set, a 64-bit first element index; if flag 0x02 is set, a minimum and a maximum, both IEEE 754 little-endian doubles (the inclusive range of valid values, for quantised reals). The column ID is its index in the list, and columns of one field go from smaller to larger IDs.

Encodings (applied inside each page, so decode page by page, never cluster by cluster): split writes all first bytes of the elements, then all second bytes, and so on; delta + split keeps the first element and stores every other as the difference from the previous, then splits; zigzag + split (signed integers only) maps x to 2x if positive and to −(2x+1) if negative, then splits. Later versions may add column types without a feature flag; old readers ignore such columns and the fields built from them, but can still tell how many elements they hold.

Deferred columns. A reader must produce synthetic pages of zero bytes for elements 0 to first element index − 1 (zero values for any supported C++ type, variants and collections included); those leading pages have no page locator. In practice deferred columns appear only in the footer's schema extension. A negative first element index means the column is deferred and suppressed: no synthetic pages exist up to and including the cluster of the first element. An unsuppressed deferred column may only belong to a field with no collection or variant ancestor (columns of nested structs can be deferred, a struct under a vector cannot, because the format does not record the sum of elements per cluster and cluster group).

Representations. A field can have several alternative column representations, told apart by the consecutive representation index starting at 0. In a cluster only one is primary; the others are suppressed. All representations have the same number of columns, with the same number of elements, and are stored in order of increasing representation index.

Alias columns

Two 32-bit integers: the physical column ID and the ID of the projected field. An alias column has no data pages and no ID of its own (nothing refers to it); the footer and page list mention only physical column IDs. Columns attach to projected fields in serialisation order (header first, then footer).

Extra type information

A record frame of a 32-bit content identifier and a 32-bit type version, then a string with the type name. Version, type name and identifier must be unique together. The only identifier is 0x00, serialised ROOT streamer information: a ROOT-streamed TList of TStreamerInfo objects combined from all fields written with the ROOT streamer; the writer sets version 0 and an empty type name, and readers should ignore it.

Footer envelope

Contents: feature flags (a reader combines them with the header's by logical or; the header's need not be repeated); the XXH3 64-bit header checksum (to check that header and footer belong together); the schema extension record frame; a list frame of cluster group record frames; a list frame of linked attribute set record frames.

Schema extension. A record frame holding the same four list frames as the end of the header (fields, columns, alias columns, extra type information), incremental to the header, possibly empty. It is read as if appended to the header: IDs continue from the largest in the header, subfields may be added to parents from the header, and columns may add representations to fields from the header.

Cluster group record frame. 64-bit minimum entry number (the smallest first entry among its clusters), 64-bit entry span (entries covered), 32-bit number of clusters (so cluster IDs stay consistent when groups are read out of order); then an envelope link to the page list envelope.

Linked attribute set record frame. 16-bit schema version major, 16-bit schema version minor, 32-bit uncompressed size of the attribute anchor (including its 8 checksum bytes); then a locator to that anchor and a string with the attribute set name.

Page list envelope

Contents: the 64-bit header checksum; a list frame of cluster summary record frames; a nested list frame of page locations.

Cluster summary: 64-bit first entry number, then 64 bits of which the low 56 are the number of entries and the high 8 are flags. Cluster IDs follow the order of the summaries, starting from the first cluster ID of the group. Flag 0x01 is reserved for sharded clusters (a future, forward-incompatible feature with its own feature flag): readers must stop when they see it and ignore other flags.

Page locations: an outer list frame with one item per cluster; each item is a list frame with one item per column (in the order of the header and extension, small to large); each of those is an inner list frame with one item per page, followed by a signed 64-bit element offset and, unless the column is suppressed, the 32-bit compression settings (the inner frame's size includes both). A page item is a 32-bit number of elements, whose sign bit C says that an XXH3 64-bit checksum of the compressed page data follows the page (the locator's size does not include it), followed by the page's locator. The uncompressed size is the number of elements times the element size, so it is not stored. Locators may share identical byte ranges but must not overlap in any other way. The element offset is needed to read an entry range without reading earlier clusters' metadata. Page items are not record frames, to save space.

Suppressed columns. A negative element offset (writers use the lowest int64) marks a suppressed column: empty page list, no compression settings. It belongs to a secondary representation that is inactive in this cluster; the count of columns and the absolute offsets are the same as for the primary representation, from which readers take the offset and element count. Each field has exactly one primary representation per cluster, and it may change between clusters.

Linked attribute sets

User-defined metadata. Each set is stored as an RNTuple whose anchor the main footer links. It may not itself link attribute sets, its alias column lists in header and footer must be empty (no projected fields), and none of its fields may have structural role 0x04. Each set has a non-empty distinct name; names that start with two underscores are reserved for implementations (a compliant writer stops users creating them; read access may be allowed).

The attribute schema version, now 1.0, describes the set's internal schema: a major change is not forward-compatible (refuse unknown major versions), a minor change adds optional fields (ignore them). The fields, in order: _rangeStart (std::uint64_t), _rangeLen (std::uint64_t; 0 is a valid empty range) and _userData, an untyped record that is the root of the user's schema; the user's top-level fields attach below it. They tie each attribute entry to a range of entries in the main RNTuple.

Mapping of C++ types to fields and columns

Type name normalisation

• Typedefs and usings are resolved, except as stated below. signed char, unsigned char and [signed|unsigned](short|int|long[ long]) become std::[u]int(8|16|32|64)_t.
• Supported stdlib types are not resolved further (std::string stays, not std::basic_string<char>).
• C arrays T[N][M] become std::array<std::array<T,M>,N>.
• volatile and const are dropped unless they appear in template arguments of user types; class, struct and enum keywords are dropped.
• Names are fully qualified, with the leading :: removed; template arguments are normalised too. For user types default template arguments are written out; optional arguments of stdlib types such as allocators are omitted.
• Integer template arguments are decimal, with suffix u only above 2⁶³. Only the whitespace C++ requires is kept.

The type alias is normalised the same way but without resolving typedefs. Example: const pair<size_t, array<class ::Event, 2>> is stored as std::pair<std::uint64_t,std::array<Event,2>> with alias std::pair<size_t,std::array<Event,2>> (64-bit).

ROOT typedefs. Bool_t, Float_t, Double_t, [U]Char_t, [U]Short_t, [U]Int_t and [U]Long[64]_t map directly to the normalised fundamental types. Exception: [U]Long64_t inside the template arguments of a user class is a plain typedef, so MyClass<std::set<Long64_t>> is stored as MyClass<std::set<std::int64_t>> with the original spelling as alias (needed to match streamer information). Double32_t is a normal typedef, normalised to double with the alias Double32_t and stored with a SplitReal32 column.

Fundamental types

A plain field with one column. Default write column (the specification marks it with an asterisk): bool Bit, std::byte Byte, char Char, int8_t Int8, uint8_t UInt8, int16_t to uint64_t the (Split) integer of the same width and signedness, float (Split)Real32, double (Split)Real64; the split variant is the default when the RNTuple is compressed and the non-split one when it is stored uncompressed. A double may also be written as Real16, (Split)Real32, Real32Trunc or Real32Quant and a float as Real16, Real32Trunc or Real32Quant. Reading is more permissive: bool, char and every integer type can be read from Bit, Char, Int8, UInt8 or any (Split) integer column of any width; float can additionally be read from (Split)Real64.

Collections and standard types

Collections have a parent column of type (Split)Index32 or (Split)Index64 holding the end offset of each entry's elements relative to the cluster. For a std::vector<float> with values {1.0}, {}, {1.0, 2.0} in three entries the index column is [1, 1, 3] and the value column [1.0, 1.0, 2.0]. Child fields are named _0, _1, and so on, unless said otherwise.

User-defined classes

Regular class or struct. Needs a dictionary; persistent members and bases must themselves be supported; transient members are marked (for example //!); not in std; empty or splittable (no custom streamer); no polymorphism, virtual inheritance or bit-fields; template arguments only integers or supported types. Stored as a record parent field without columns. Members are subfields named like the C++ member; direct base classes are subfields named :_0, :_1, and so on.

Class with a collection proxy. A non-associative one is stored like std::vector<T>; an associative one like std::map<K,V> (the specification says proxy-based associative collections are in the format but not yet implemented in ROOT's reader and writer).

SoA class. A class marked in its dictionary as the struct-of-arrays layout of a record type T is stored like std::vector<T> (child _0); the field's type name is the SoA class, its type checksum and version are stored, and field flag 0x08 is set.

ROOT::RNTupleCardinality<SizeT> (std::uint32_t or std::uint64_t) is a field with one (Split)Index column that presents the offsets as lengths: value i is index i minus index i−1, with 0 taken for the first element of a cluster (index [1, 1, 3] gives [1, 0, 2]). It is meant as a projected field that only reads a collection's size.

ROOT streamed types. A field with structural role 0x04 holds objects serialised by the ROOT streamer: a (Split)Index column and a Byte column, like a collection of std::byte. Any type TClass supports is allowed; the template-argument restrictions still apply.

Untyped collections and records have the collection or record role and an empty type name; only top-level fields and direct subfields of untyped fields may be untyped. Otherwise they are stored like std::vector and like a user class.

Limits, names and defaults

Names. An RNTuple name or field name cannot be empty when persisted. Allowed: Unicode as UTF-8, except all control codes (newline and tab included), full stop, space, backslash and slash.

Writer defaults (RNTupleWriteOptions): approximate zipped cluster 128 MiB, maximum unzipped cluster 1280 MiB, maximum unzipped page 1 MiB.

Terms and compatibility

A page is a segment of a column and the unit of compression; a cluster is a non-empty set of pages holding all the data for an entry range (empty clusters are forbidden); a cluster group is a non-empty set of clusters with one page list envelope, so page lists can be streamed for very large data sets. A column is a vector of fixed-bit-length elements; a field is a serialised C++ type with zero or more columns holding its own data (subfields have their own). Length means size in memory uncompressed, size means size on disk.

Readers must read files of earlier versions of the same epoch. For newer versions of the epoch they must: ignore unknown trailing data in the anchor, envelopes and frames (still verifying checksums over known and unknown content); ignore unknown column, cluster and field flags and unknown extra-type-information IDs; ignore the whole top-level field that holds an unknown column type, field type, field version or structure, together with projected fields and alias columns that depend on it; refuse to read on an unknown feature flag. Writers must set the feature flags for any feature that stops older readers, and write the version they support in the anchor whether or not they use every feature of it.