A sample entry in stsd names the codec; the boxes inside it configure the
decoder. The sample entry layout comes
first, then these boxes follow to the end of the entry. All numbers are big-endian. None of
these layouts is in a specification this reference could read. They were worked out from
memory and checked by reading the boxes of files that FFmpeg 8 writes (AAC, ALAC, AC-3,
E-AC-3, MP3, Opus, FLAC, H.264, HEVC, AV1 and VP9 in MP4, and PCM in MOV): in every one,
the layouts below account for every byte of the box
(parser-test/test-mp4-built.mjs). Fields that FFmpeg never writes, such as a
second sequence parameter set list, are from memory only.
esds: elementary stream descriptor
A full box whose payload is an MPEG-4 descriptor (ISO/IEC 14496-1). A descriptor is a tag
byte, a length in 1 to 4 bytes of 7 bits each (the top bit says another byte follows), and
a body; descriptors nest. Used by mp4a (AAC, MP3 and others) and
mp4v (MPEG-4 Visual).
| Tag | Descriptor | Body |
|---|---|---|
03 | ES_Descriptor | ES_ID (2, zero in MP4); flags (1): streamDependenceFlag, URL_Flag, OCRstreamFlag (1 bit each), streamPriority (5 bits); then dependsOn_ES_ID (2), the URL (length byte and text) and OCR_ES_Id (2) when flagged; then the descriptors below. |
04 | DecoderConfigDescriptor | objectTypeIndication (1); streamType (6 bits), upStream (1), reserved (1); bufferSizeDB (3); maxBitrate (4); avgBitrate (4); then DecoderSpecificInfo. |
05 | DecoderSpecificInfo | Decoder data. For AAC, the AudioSpecificConfig (ISO/IEC 14496-3). |
06 | SLConfigDescriptor | predefined (1): 2 in MP4 files. 0 would mean a custom configuration follows. |
Other tags the viewer names but does not read: 01 ObjectDescriptor, 02 InitialObjectDescriptor, 07 ContentIdentificationDescriptor, 08 SupplementaryContentIdentificationDescriptor, 09 IPI_DescrPointer, 0A IPMP_DescriptorPointer, 0B IPMP_Descriptor, 0C QoS_Descriptor, 0D RegistrationDescriptor, 0E ES_ID_Inc, 0F ES_ID_Ref, 10 MP4_IOD, 11 MP4_OD, 12 IPI_DescrPointerRef, 13 ExtendedProfileLevelDescriptor, 14 ProfileLevelIndicationIndexDescriptor.
streamType: 01 ObjectDescriptorStream, 02 ClockReferenceStream, 03 SceneDescriptionStream, 04 VisualStream, 05 AudioStream, 06 MPEG7Stream, 07 IPMPStream, 08 ObjectContentInfoStream, 09 MPEGJStream, 0A Interaction, 0B IPMPToolStream, 0C FontDataStream, 0D StreamingText (MP4 Registration Authority).
objectTypeIndication values that matter in MP4 (MP4 Registration Authority): 20 MPEG-4 Visual, 21 H.264, 22 H.264 parameter sets, 23 H.265, 40 MPEG-4 Audio (AAC, and every other 14496-3 object type: the AudioSpecificConfig says which), 60 to 65 MPEG-2 Visual profiles, 66 to 68 MPEG-2 AAC (Main, LC, SSR), 69 MPEG-2 audio (MP3), 6A MPEG-1 Visual, 6B MPEG-1 audio (MP3), 6C JPEG, 6D PNG, 6E JPEG 2000, A3 VC-1, A4 Dirac, A7 DRA, A8 G.719, A9 to AC DTS-HD, AD Opus, AF Auro-Cx, B1 VP9, B2 and B3 DTS-UHD, E1 13K voice, FF none. A5 (AC-3), A6 (E-AC-3) and AE (AC-4) are withdrawn: those codecs use their own sample entries.
avcC: AVC configuration
The AVCDecoderConfigurationRecord of ISO/IEC 14496-15, inside avc1,
avc2, avc3 and avc4. The samples are H.264 NAL units,
each preceded by a length of lengthSizeMinusOne + 1 bytes (not start codes).
| Size | Name | Meaning |
|---|---|---|
| 1 | configurationVersion | 1. |
| 1 | AVCProfileIndication | profile_idc of the stream: 66 Baseline, 77 Main, 88 Extended, 100 High, 110 High 10, 122 High 4:2:2, 244 High 4:4:4 Predictive, 44 CAVLC 4:4:4 Intra, and the multiview and scalable profiles. |
| 1 | profile_compatibility | The constraint_set flags and reserved bits of the SPS. |
| 1 | AVCLevelIndication | level_idc, ten times the level (41 is level 4.1). |
| 6 bits + 2 bits | reserved, lengthSizeMinusOne | Reserved bits all ones; the NAL unit length size minus 1. |
| 3 bits + 5 bits | reserved, numOfSequenceParameterSets | Reserved bits all ones; the count. |
| each | sequenceParameterSetLength (2), the SPS NAL unit | With its one-byte NAL header. |
| 1 | numOfPictureParameterSets | The count; then each as a 2-byte length and the NAL unit. |
| For profiles 100, 110, 122, 244, 44, 83, 86, 118, 128, 138, 139, 134 and 135 only: | ||
| 6 + 2 bits | reserved, chroma_format | 0 monochrome, 1 4:2:0, 2 4:2:2, 3 4:4:4. |
| 5 + 3 bits | reserved, bit_depth_luma_minus8 | |
| 5 + 3 bits | reserved, bit_depth_chroma_minus8 | |
| 1 | numOfSequenceParameterSetExt | The count; then each as a 2-byte length and the NAL unit. |
hvcC: HEVC configuration
The HEVCDecoderConfigurationRecord, inside hvc1 and hev1 (and
the extractor and aggregator variants). Samples are length-prefixed NAL units.
| Size | Name | Meaning |
|---|---|---|
| 1 | configurationVersion | 1. |
| 2 + 1 + 5 bits | general_profile_space, general_tier_flag, general_profile_idc | Profile space (0); tier (0 Main, 1 High); profile (1 Main, 2 Main 10, 3 Main Still Picture, 4 format range extensions, 5 high throughput, 6 multiview main, 7 scalable main, 8 3D main, 9 screen content coding, 10 scalable format range extensions, 11 high throughput screen content coding). |
| 4 | general_profile_compatibility_flags | One bit for each profile. |
| 6 | general_constraint_indicator_flags | 48 flag bits. |
| 1 | general_level_idc | Thirty times the level (93 is level 3.1). |
| 4 + 12 bits | reserved, min_spatial_segmentation_idc | Reserved bits all ones. |
| 6 + 2 bits | reserved, parallelismType | 0 mixed or unknown, 1 slice, 2 tile, 3 wavefront. |
| 6 + 2 bits | reserved, chromaFormat | As avcC. |
| 5 + 3 bits | reserved, bitDepthLumaMinus8 | |
| 5 + 3 bits | reserved, bitDepthChromaMinus8 | |
| 2 | avgFrameRate | Frames per 256 seconds; 0 unspecified. |
| 2 + 3 + 1 + 2 bits | constantFrameRate, numTemporalLayers, temporalIdNested, lengthSizeMinusOne | constantFrameRate: 0 not constant, 1 constant, 2 constant in each temporal layer. |
| 1 | numOfArrays | The number of NAL unit arrays; each is: |
| 1 + 1 + 6 bits | array_completeness, reserved, NAL_unit_type | 32 VPS, 33 SPS, 34 PPS, 39 prefix SEI, 40 suffix SEI. |
| 2 | numNalus | Then for each NAL unit: nalUnitLength (2) and the NAL unit. |
av1C: AV1 configuration
The AV1CodecConfigurationRecord (AV1 Codec ISO Media File Format Binding), inside
av01.
| Bits | Name | Meaning |
|---|---|---|
| 1 + 7 | marker, version | Both 1 (the byte is 0x81). |
| 3 + 5 | seq_profile, seq_level_idx_0 | Profile 0 Main, 1 High, 2 Professional; the level index (0 is level 2.0, 1 is 2.1, 4 is 3.0). |
| 1 each | seq_tier_0, high_bitdepth, twelve_bit, monochrome, chroma_subsampling_x, chroma_subsampling_y | Tier; bit depth above 8; with high_bitdepth, 12-bit rather than 10; monochrome; the subsampling. |
| 2 | chroma_sample_position | 0 unknown, 1 vertical, 2 co-located. |
| 3 + 1 + 4 | reserved, initial_presentation_delay_present, initial_presentation_delay_minus_one | The delay, in samples minus 1, when present. |
| to the end | configOBUs | Open Bitstream Units: normally the sequence header OBU. |
Samples are sequences of OBUs. An OBU header is a byte: forbidden (1 bit), type (4), extension flag (1), has-size flag (1), reserved (1); then an optional extension byte, then when has-size is set a length in LEB128. Types: 1 sequence header, 2 temporal delimiter, 3 frame header, 4 tile group, 5 metadata, 6 frame, 7 redundant frame header, 8 tile list, 15 padding.
vpcC: VP codec configuration
A full box (VP Codec ISO Media File Format Binding), inside vp08 and
vp09.
| Size | Name | Meaning |
|---|---|---|
| 1 | profile | 0 to 3. |
| 1 | level | Ten times the level. |
| 4 + 3 + 1 bits | bitDepth, chromaSubsampling, videoFullRangeFlag | 8, 10 or 12; 0 4:2:0 vertical, 1 4:2:0 co-located, 2 4:2:2, 3 4:4:4; 1 for full range. |
| 1 | colourPrimaries | ISO/IEC 23091-2 code points. |
| 1 | transferCharacteristics | |
| 1 | matrixCoefficients | |
| 2 | codecInitializationDataSize | 0 for VP8 and VP9; the data follows. |
Audio: dac3, dec3, dOps, dfLa, alac, pcmC, srat
| Box | In entry | Fields |
|---|---|---|
dac3 | ac-3 | Three bytes: fscod (2 bits: 0 48 kHz, 1 44.1 kHz, 2 32 kHz), bsid (5), bsmod (3), acmod (3), lfeon (1), bit_rate_code (5), reserved (5). acmod: 0 dual mono (2 channels), 1 C, 2 L R, 3 L C R, 4 L R S, 5 L C R S, 6 L R Ls Rs, 7 L C R Ls Rs; add one channel when lfeon is set. |
dec3 | ec-3 | data_rate (13 bits, kbit/s), num_ind_sub (3 bits, minus 1); then for each independent substream: fscod (2), bsid (5), reserved (1), asvc (1), bsmod (3), acmod (3), lfeon (1), reserved (3), num_dep_sub (4), then chan_loc (9 bits) if num_dep_sub is not 0, else one reserved bit. Optional extension bytes follow (the Dolby Atmos flag and complexity index). |
dOps | Opus | Version (1, 0), OutputChannelCount (1), PreSkip (2), InputSampleRate (4), OutputGain (2, signed, Q7.8 dB), ChannelMappingFamily (1); if not 0: StreamCount (1), CoupledCount (1), ChannelMapping (one byte for each output channel). All big-endian, unlike the Ogg Opus header. |
dfLa | fLaC | A full box holding FLAC metadata blocks: each a header byte (last-block flag in the top bit, block type in 7 bits: 0 STREAMINFO, 1 PADDING, 2 APPLICATION, 3 SEEKTABLE, 4 VORBIS_COMMENT, 5 CUESHEET, 6 PICTURE), a 3-byte length and the data. The first is STREAMINFO (34 bytes). |
alac | alac | A full box holding the ALAC magic cookie: frameLength (4), compatibleVersion (1), bitDepth (1), pb (1), mb (1), kb (1), numChannels (1), maxRun (2), maxFrameBytes (4), avgBitRate (4), sampleRate (4); then optionally a 24-byte chan box: size, type, version and flags, channel layout tag (4), two reserved words. |
pcmC | ipcm, fpcm | A full box: format_flags (1; bit 0 set for little-endian), PCM_sample_size (1: 16, 24, 32 or 64 bits). |
srat | audio, entry version 1 | A full box: sampling_rate (4), in Hz, for rates the 16.16 field of the entry cannot hold. |
Picture: pasp, clap, colr, fiel, btrt
| Box | Fields |
|---|---|
pasp | hSpacing (4), vSpacing (4): the shape of a pixel. |
clap | cleanApertureWidthN, cleanApertureWidthD, cleanApertureHeightN, cleanApertureHeightD, horizOffN, horizOffD, vertOffN, vertOffD (4 each; the offsets are signed). |
colr | colour_type (4): nclx (colour_primaries (2), transfer_characteristics (2), matrix_coefficients (2), full_range_flag (1 bit) and 7 reserved bits), nclc (the first three only, the QuickTime form), rICC or prof (an ICC profile follows). |
fiel | QuickTime: fields (1: 1 progressive, 2 interlaced), detail (1: for interlaced, 1 top first, 6 bottom first, 9 and 14 for the cross-stored orders). |
gama | QuickTime: gamma (4, 16.16). |
btrt | bufferSizeDB (4), maxBitrate (4), avgBitrate (4). In any sample entry. |
colour_primaries, transfer_characteristics and matrix_coefficients use the code points of ISO/IEC 23091-2 (the same as H.273): primaries 1 BT.709, 5 BT.601 625, 6 BT.601 525, 9 BT.2020, 12 Display P3; transfer 1 BT.709, 13 sRGB, 16 PQ, 18 HLG; matrix 0 identity, 1 BT.709, 5 and 6 BT.601, 9 BT.2020 non-constant luminance.
QuickTime sound extensions: wave, frma, enda, chan
QuickTime sound description versions 1 and 2 (see
Sound descriptions) keep the codec's
own boxes in a wave box: frma (data_format, 4), then
enda (a 16-bit flag: 1 little-endian) for PCM, esds or
alac for the codec, a four-byte-zero mp4a atom before
esds, and an eight-byte zero terminator. chan (a full box):
mChannelLayoutTag (4), mChannelBitmap (4), mNumberChannelDescriptions (4), then that many
descriptions of 20 bytes: label (4), flags (4), three 32-bit floats.
Text and metadata entries
| Entry | Fields after the entry header |
|---|---|
tx3g | 3GPP timed text: see 3GPP sample entries. Child ftab: entry-count (2); each font: font-ID (2), font-name-length (1), the name. |
stpp | namespace, schema_location, auxiliary_mime_types: three zero-ended strings. |
sbtt, stxt, mett | content_encoding, mime_format: two zero-ended strings. (sbtt may have a txtC child.) |
metx | content_encoding, namespace, schema_location: three zero-ended strings. |
wvtt | Child boxes vttC (the WebVTT header text) and vlab (a source label). |
urim | Child uri (a full box with a zero-ended URI) and optionally uriI. |
tmcd | QuickTime timecode: see Other media. |