This page follows the
Arrow Columnar Format
specification and its FlatBuffers schemas in
format/
(Message.fbs, Schema.fbs, File.fbs), all read
2026-11-12. Numbers are little-endian unless the schema says otherwise.
Encapsulated messages
| Field | Size | Meaning |
|---|---|---|
| Continuation | 4 | FF FF FF FF. Files written before Arrow 0.15 leave it out. |
| Metadata size | 4 | Size of the FlatBuffers Message plus padding, a multiple of 8. |
| Message | … | FlatBuffers Message: version, header (a union), bodyLength, custom_metadata. |
| Body | bodyLength | The buffers, each aligned to 8 bytes (64 recommended). |
A stream is a Schema message, then dictionary batches and record
batches, ended by FF FF FF FF 00 00 00 00. Header types:
Schema, DictionaryBatch, RecordBatch,
Tensor, SparseTensor. Metadata versions run from V1 (0) to
V5 (4), the current one.
File format
ARROW1and 2 zero bytes.- A stream as above (the schema, then batches, then the end marker).
- The footer, a FlatBuffers
Footer:version,schema,dictionariesandrecordBatchesas lists ofBlock(offsetint64,metaDataLengthint32,bodyLengthint64),custom_metadata. - The footer’s size, 4 bytes.
ARROW1.
The file format does not allow dictionary replacement: each dictionary may be sent once, plus deltas.
Schema and fields
Schema:endianness(0 little, 1 big),fields,custom_metadata,features(1 dictionary replacement, 2 compressed body).Field:name,nullable,type,dictionary(id,indexType,isOrdered,dictionaryKind),children,custom_metadata(key and value strings, such asARROW:extension:namefor extension types).
| ID | Type | Parameters |
|---|---|---|
| 1 | Null | |
| 2 | Int | bitWidth (8, 16, 32, 64), is_signed |
| 3 | FloatingPoint | precision: 0 half, 1 single, 2 double |
| 4 | Binary | 32-bit offsets |
| 5 | Utf8 | 32-bit offsets |
| 6 | Bool | Bit-packed |
| 7 | Decimal | precision, scale, bitWidth (32, 64, 128, 256) |
| 8 | Date | unit: 0 day (int32), 1 millisecond (int64) |
| 9 | Time | unit (0 s, 1 ms, 2 µs, 3 ns), bitWidth |
| 10 | Timestamp | unit, timezone (absent for local times) |
| 11 | Interval | unit: 0 year-month, 1 day-time, 2 month-day-nano |
| 12 | List | One child; 32-bit offsets |
| 13 | Struct | Children |
| 14 | Union | mode (0 sparse, 1 dense), typeIds |
| 15 | FixedSizeBinary | byteWidth |
| 16 | FixedSizeList | listSize |
| 17 | Map | keysSorted; a list of key-value structs |
| 18 | Duration | unit |
| 19 | LargeBinary | 64-bit offsets |
| 20 | LargeUtf8 | 64-bit offsets |
| 21 | LargeList | 64-bit offsets |
| 22 | RunEndEncoded | Run ends and values children |
| 23 | BinaryView | 16-byte views and variadic data buffers |
| 24 | Utf8View | As BinaryView |
| 25 | ListView | Offsets and sizes, 32-bit |
| 26 | LargeListView | Offsets and sizes, 64-bit |
Record batches
RecordBatch:length(rows),nodes(oneFieldNodeper field, depth first:length,null_count),buffers(offsetandlengthin the body),compression,variadicBufferCounts(for view types).DictionaryBatch:id,data(a record batch with one column),isDelta.- Buffers per array: a validity bitmap (least significant bit first; may be empty when there are no nulls), then offsets and/or values as the type needs.
BodyCompression:codec0 LZ4 frame, 1 Zstandard;method0 buffer by buffer. Each compressed buffer starts with its uncompressed length as int64, or −1 when it was left uncompressed.