.AVRO

Introduction

Apache Avro is a binary data format from the Hadoop project, first released in 2009. Records are described by a schema written in JSON, and the binary data holds only the values, in schema order, with no field names or type bytes. That makes it compact, but the schema is needed to read it.

An .avro file is an object container file: it carries its schema in the header, so any Avro reader can open it. Avro is widely used with Kafka and in data lakes, where readers can also handle data written with an older or newer version of the schema.

Structure

The file starts with the four bytes Obj and 01. Then comes a metadata map, which holds the schema as JSON under avro.schema and the compression codec under avro.codec (null, deflate, snappy, zstandard and others), and a random 16-byte sync marker.

The rest is a run of data blocks. Each gives the number of records and the size in bytes, then the records, compressed with the codec, then the sync marker again, which lets a reader find block boundaries when it starts in the middle of a file. Integers and lengths are written as variable-length zig-zag numbers.

Links