.ARROW

Introduction

Apache Arrow is a standard way of laying out tables of data in memory, column by column, shared by many data tools so that they can pass data to each other without converting it. The Arrow IPC file format writes that same memory layout to disk, so a file can be memory-mapped and used at once, with no parsing.

The format is also known as Feather. Feather version 1 was a separate, simpler format made in 2016 for exchanging data frames between Python and R; Feather version 2 is the Arrow IPC file format, optionally compressed. Files are named .arrow or .feather. For long-term storage, Parquet is usually preferred because it compresses better.

Structure

The file starts with the six bytes ARROW1 and two bytes of padding, and ends with ARROW1. Between them come the schema, any dictionaries used to encode repeated values, and the record batches: groups of rows, each holding one buffer of values per column, aligned to 8 bytes. The messages that describe them are encoded with FlatBuffers.

A footer before the closing magic repeats the schema and gives the position of every record batch and dictionary, so a reader can go straight to any batch. Its length is the 32-bit number just before the final ARROW1.

A related stream format, sometimes saved as .arrows, has no magic and no footer and is read from start to end.

Links