MP4 metadata and items

Metadata lives in meta boxes (at the file, movie or track level), in udta user data, and, for images and other items, in the item boxes of HEIF. All numbers are big-endian. These layouts come from memory of ISO/IEC 14496-12 and ISO/IEC 23008-12, neither of which could be read, from the MP4 Registration Authority's box lists, and from Apple's QuickTime File Format pages (for ilst and keys). The ones FFmpeg writes (meta, hdlr, ilst, data, chpl, a QuickTime chapter track) were checked against files it made, and 15 HEIC files on the test machine were read without a gap; the rest were not tested.

meta, hdlr, pitm

meta in an ISO file is a full box (version and flags, then boxes). In QuickTime it is a plain box that starts with the hdlr box; a reader tells them apart by whether the type hdlr is at offset 4 or offset 8 of the payload. hdlr says what kind of metadata follows (mdta or mdir for iTunes and QuickTime keys, pict for HEIF items, ID32 for ID3, mp7t and mp7b for MPEG-7, null). pitm (full box): item_ID (2 in version 0, otherwise 4), the primary item. meco holds further meta boxes; mere (full box) relates two of them: two handler types (4 each) and a relation (1: 1 unrelated, 2 complementary, 3 the second is a more specific view, 4 alternatives).

ilst, keys and data

ilst holds one box for each item. In iTunes files its type is a four-character name; in QuickTime metadata it is a 1-based index into the keys box. Each item holds a data box for each value, and free-form iTunes items (----) hold mean and name (full boxes of UTF-8 text) before it.

Box	Fields
`keys`	Full box. entry_count (4); each key: key_size (4, including these 8 bytes), key_namespace (4: `mdta` for reverse-DNS names, `udta` for user-data atom types), key_value.
`data`	type indicator (4: one byte of type set, 0 for the well-known types, then 24 bits of type), country (2), language (2), the value to the end. Country: 0 default, 1 to 255 an index into `ctry`, otherwise two ASCII letters. Language: 0 default, 1 to 255 an index into `lang`, otherwise a packed ISO 639-2/T code.
`mean`, `name`	Full boxes; the text fills the rest.
`itif`	QuickTime: a full box with item_ID (4).

Well-known types: 0 reserved, 1 UTF-8, 2 UTF-16 big-endian, 3 S/JIS, 4 UTF-8 sort, 5 UTF-16 sort, 13 JPEG, 14 PNG, 21 big-endian signed integer (1 to 4 bytes), 22 unsigned, 23 float32, 24 float64, 27 BMP, 28 QuickTime metadata atom, 65 8-bit signed, 66 and 67 16- and 32-bit signed, 70 PointF32, 71 DimensionsF32, 72 RectF32, 74 64-bit signed, 75 to 78 8-, 16-, 32- and 64-bit unsigned, 79 AffineTransformF64 (Apple QuickTime File Format). In iTunes tags, trkn and disk hold 2 reserved bytes, the number (2), the total (2) and 2 more reserved bytes; gnre holds an ID3 genre number plus one (2 bytes).

Item names that iTunes and its tools use (these are conventions, not in an ISO specification): ©nam title, ©ART artist, ©alb album, aART album artist, ©gen and gnre genre, ©day date, trkn track, disk disc, tmpo tempo, cpil compilation, pgap gapless, covr cover art, ©wrt composer, ©too encoder, ©cmt comment, desc and ldes descriptions, ©lyr lyrics, ©grp grouping, cprt copyright, stik media kind, rtng content rating, hdvd HD flag, soal, soaa, soar, sonm, soco, sosn sort names, tvsh, tvsn, tvep, tves, tvnn television, purd purchase date, pcst, catg, keyw, egid, purl podcasts, apID, cnID, atID, plID, geID, sfID, ownr store identifiers, ©mvn, ©mvi, ©mvc, shwm movements. The viewer does not read the contents of the item types beyond showing them as text, numbers or an image.

Items: iinf, infe, iloc, iref, iprp, ipco, ipma, grpl

Box	Fields
`iinf`	Full box: entry_count (2, or 4 from version 1), then `infe` boxes.
`infe`	Full box; flag 1: item is hidden. Versions 0 and 1: item_ID (2), item_protection_index (2), item_name (zero-ended), content_type, and optionally content_encoding. Versions 2 and 3: item_ID (2, or 4 in version 3), item_protection_index (2), item_type (4), item_name; then for `mime` the content_type and optionally content_encoding, for `uri` an item_uri_type.
`iloc`	Full box. offset_size and length_size (4 bits each), base_offset_size and index_size (4 bits each; the second is reserved in version 0): each size is 0, 4 or 8 bytes. item_count (2, or 4 in version 2). Each item: item_ID (2, or 4), in versions 1 and 2 reserved (12 bits) and construction_method (4 bits: 0 file offset, 1 `idat` offset, 2 item offset), data_reference_index (2), base_offset, extent_count (2); each extent: extent_index (versions 1 and 2, when index_size is not 0), extent_offset, extent_length.
`iref`	Full box of reference boxes, each with the reference type as its type: from_item_ID (2, or 4 in version 1), reference_count (2), to_item_ID (2 or 4) for each. Types: `auxl` auxiliary image, `base` pre-derived base, `cdsc` describes, `dimg` derived image, `dpnd` coding dependency, `eroi` region, `exbl` scalable, `mask` region mask, `pred` predictively coded, `prem` pre-multiplied, `sato` sample transform, `tbas` tile base, `thmb` thumbnail, and others in the registry.
`iprp`, `ipco`	Containers: `ipco` holds the property boxes, which `ipma` refers to by 1-based index.
`ipma`	Full box; flag 1: wide property indexes. entry_count (4); each: item_ID (2, or 4 from version 1), association_count (1), then essential (1 bit) and property_index (7 bits, or 15 with flag 1) for each.
`grpl`	Container of entity group boxes (type = grouping type): full box with group_id (4), num_entities_in_group (4), entity_id (4) each. Types in the registry include `altr` alternatives, `ster` stereo, `brst` burst, `slid` slideshow, `pano` panorama.

Item properties

Box	Fields
`ispe`	Full box: image_width (4), image_height (4).
`pixi`	Full box: num_channels (1), bits_per_channel (1) each.
`irot`	reserved (6 bits), angle (2 bits): the anticlockwise rotation in units of 90 degrees.
`imir`	reserved (7 bits), axis (1 bit): 0 vertical axis (left and right swap), 1 horizontal axis (top and bottom swap).
`auxC`	Full box: aux_type (zero-ended URN), then aux_subtype to the end.
`rloc`	Full box: horizontal_offset (4), vertical_offset (4).
`clli`	max_content_light_level (2), max_pic_average_light_level (2), in cd/m².
`mdcv`	Three display primaries and the white point (x and y, 2 bytes each, units of 0.00002), max_display_mastering_luminance (4) and min_display_mastering_luminance (4), units of 0.0001 cd/m².
`colr`, `pasp`, `clap`, `hvcC`, `av1C`, `avcC`	As in Codec configuration boxes.

Item types in infe (registry): av01, avc1, hvc1, hvt1, vvc1, evc1, jpeg, j2k1, jxs1, lhv1 (coded images), grid, iden, iovl (derived images), Exif, mime, mski, rgan, mint, unci, it35, nnr1, auvd, auxi. The viewer lists the items and their types; the item data (a grid description, an Exif block, the image itself) is not decoded.

User data: udta

udta holds boxes of text and other information. In QuickTime files its children are atoms whose type starts with © (0xA9) and hold a 2-byte text size, a 2-byte language and the text (©nam name, ©ART artist, ©cmt comment, ©day date, ©gen genre, ©too encoding tool, ©xyz GPS location and so on); a list of such atoms ends with a 32-bit zero. The ISO and 3GPP boxes below are full boxes.

Box	Fields
`cprt`, `titl`, `dscp`, `perf`, `auth`, `gnre`	pad (1 bit) and language (15 bits: three letters of ISO 639-2/T, each minus 0x60), then the text: UTF-8 ended by zero, or UTF-16 starting with a byte order mark and ended by a zero word.
`albm`	As `titl`, then optionally track_number (1).
`yrrc`	language (2), year (2).
`kywd`	language (2), KeywordCnt (1); each keyword: size (1) and the text with its zero.
`rtng`	RatingEntity (4), RatingCriteria (4), language (2), RatingInfo (text).
`clsf`	ClassificationEntity (4), ClassificationTable (2), language (2), ClassificationInfo (text).
`loci`	language (2), name (zero-ended), role (1: 0 shooting location, 1 real location, 2 fictional location), longitude, latitude and altitude (each 4, 16.16 fixed point), astronomical_body (zero-ended), additional_notes (zero-ended).
`kind`	schemeURI (zero-ended), value (zero-ended).
`elng`	extended_language: a BCP 47 tag, zero-ended.
`tsel`	switch_group (4, signed), then attribute fourccs to the end (`lang`, `bitr`, `frar`, `scsz`, `cdec`, `mode`, …).
`ID32`	language (2), then an ID3v2 tag.
`xml` , `bxml`	An XML document, or its binary encoding.
`chpl`	Nero chapter list; not in the MP4 registry. Full box: reserved (4), chapter_count (1); each chapter: start_time (8, in units of 100 ns), title length (1), the title in UTF-8. The layout is the one FFmpeg reads and writes.

The 3GPP boxes are specified in 3GPP asset metadata. Boxes in the registry's user-data list that this reference does not describe field by field: ades, alou, angl, clfn, clid, cmid, cmnm, coll, date, hinf, hnti, ludt, manu, modl, orie, prmd, reel, scen, shot, slno, strk, swre, thmb, tlou, urat.

Track references and track groups

tref holds one box for each reference type; its payload is a list of 4-byte track IDs (not a full box). Types in the registry: chap (QuickTime chapter list; the referenced track is a text track whose samples are a 2-byte length and the chapter title), tmcd (time code), cdsc (describes the track), hint (hint to media), subt (subtitles), thmb (thumbnail), font, sync, dpnd, ipir, mpod, auxl, vdep, vplx, scal, sbas, tbas, sabt, avcp, adda, adrc, aest, cdep, scpt, ssrc, rndr and the V3C and VVC ones in the registry. trgr holds track group boxes, each a full box with a track_group_id (4); the type is the group type (msrc, ster, alte, cstg, 2dsr and others).