.H5

Introduction

HDF5 stores large amounts of numerical data, such as measurements, simulation output and images, organised like a file system inside one file. It began at the National Center for Supercomputing Applications and is now maintained by The HDF Group. HDF5 appeared in 1998 as a redesign of the earlier HDF4.

Several other formats are HDF5 underneath: NetCDF-4 climate data, MATLAB’s version 7.3 .mat files and the .h5 model files saved by Keras. Files use .h5 or .hdf5.

Software

• HDFView, The HDF Group’s browser and editor (Windows, macOS, Linux)
• The HDF5 library and tools, including h5dump and h5ls
• h5py, for Python
• MATLAB (Windows, macOS, Linux)
• Panoply, NASA’s viewer for gridded data (Windows, macOS, Linux)

Structure

An HDF5 file holds groups, which work like folders, and datasets, which are arrays of any number of dimensions with a data type such as 32-bit float. Both can carry named attributes for metadata. Datasets can be stored whole or in chunks, and chunks can be compressed.

The file starts with an 8-byte signature, 89 48 44 46 0D 0A 1A 0A (\211HDF\r\n\032\n). It can also sit at byte 512, 1024, 2048 or any later power of two, leaving room for a user block before it. The superblock follows the signature: its version, the sizes used for offsets and lengths in this file, and the address of the root group. Every other object is found by following addresses from there.