.PARQUET

Introduction

Apache Parquet is a file format for large tables that stores data by column rather than by row. Twitter and Cloudera created it in 2013 for the Hadoop ecosystem, and it is now the usual format for data lakes and analytics tools.

Keeping each column together means a query that needs a few columns reads only those, and values of one type side by side compress well. Each file carries its own schema and statistics, such as the minimum and maximum of each column, so readers can skip parts that cannot match a query.

Structure

A Parquet file starts and ends with the four bytes PAR1. Between them the rows are split into row groups. Within a row group each column is stored as a column chunk, made of pages of encoded and usually compressed values (with Snappy, Gzip, Zstandard or others).

The footer at the end holds the file metadata: the schema, the row groups, where each column chunk starts and its statistics. It is encoded with Apache Thrift’s compact protocol. A reader starts at the end: the last eight bytes are the footer’s length, as a 32-bit little-endian number, and PAR1.

Links