H.264 SEI messages

Supplemental enhancement information, NAL unit type 6, carries extra data about the video that a decoder does not need in order to decode: timing, recovery points, timecodes, film grain, HDR metadata, encoder notes and more.

Framing

A SEI NAL unit is a series of messages followed by trailing bits. Each message starts with two values, each written as a run of FF bytes, then a final byte:

Field	Coding	Description
`payloadType`	a run of `FF` bytes, then one byte	the sum of all the bytes
`payloadSize`	the same	payload length in bytes
`sei_payload`		that many bytes

So a type of 300 is stored as FF 2D (255 + 45). The NAL unit’s data is subject to emulation prevention.

Payload types

The numbers form one register shared by H.264, H.265 and H.266, and this is the list kept by FFmpeg. Not every number is used by H.264.

Type	Message
0	buffering period
1	picture timing
2	pan scan rectangle
3	filler payload
4	user data registered by ITU-T T.35
5	user data unregistered
6	recovery point
7	decoded reference picture marking repetition
8	spare picture
9	scene information
10	sub-sequence information
11	sub-sequence layer characteristics
12	sub-sequence characteristics
13	full-frame freeze
14	full-frame freeze release
15	full-frame snapshot
16	progressive refinement segment start
17	progressive refinement segment end
18	motion-constrained slice group set
19	film grain characteristics
20	deblocking filter display preference
21	stereo video information
22	post-filter hint
23	tone mapping information
24	scalability information
25	sub-picture scalable layer
26	non-required layer representation
27	priority layer information
28	layers not present
29	layer dependency change
30	scalable nesting
31	base layer temporal HRD
32	quality layer integrity check
33	redundant picture property
34	temporal level zero dependency representation index
35	temporal level switching point
36	parallel decoding information
37	MVC scalable nesting
38	view scalability information
39	multiview scene information
40	multiview acquisition information
41	non-required view component
42	view dependency change
43	operation points not present
44	base view temporal HRD
45	frame packing arrangement
46	multiview view position
47	display orientation
48	MVCD scalable nesting
49	MVCD view scalability information
50	depth representation information
51	three-dimensional reference displays information
52	depth timing
53	depth sampling information
54	constrained depth parameter set identifier
56	green metadata
128	structure of pictures information
129	active parameter sets
130	decoding unit information
131	temporal sub-layer zero index
132	decoded picture hash
133	scalable nesting (H.265)
134	region refresh information
135	no display
136	time code
137	mastering display colour volume
138	segmented rectangular frame packing arrangement
139	temporal motion-constrained tile sets
140	chroma resampling filter hint
141	knee function information
142	colour remapping information
143	deinterlaced field identification
144	content light level information
145	dependent RAP indication
146	coded region completion
147	alternative transfer characteristics
148	ambient viewing environment
149	content colour volume
150	equirectangular projection
151	cubemap projection
152	fisheye video information
154	sphere rotation
155	region-wise packing
156	omnidirectional viewport
157	regional nesting
158	MCTS extraction information sets
159	MCTS extraction information nesting
160	layers not present (H.265)
161	inter-layer constrained tile sets
162	BSP nesting
163	BSP initial arrival time
164	sub-bitstream property
165	alpha channel information
166	overlay information
167	temporal motion vector prediction constraints
168	frame-field information
176	three-dimensional reference displays information (H.265)
177	depth representation information (H.265)
178	multiview scene information (H.265)
179	multiview acquisition information (H.265)
180	multiview view position (H.265)
181	alternative depth information
200	SEI manifest
201	SEI prefix indication
202	annotated regions
203	subpicture level information
204	sample aspect ratio information

Buffering period (type 0)

Name	Coding	Condition
`seq_parameter_set_id`	`ue(v)`	0 to 31
`initial_cpb_removal_delay[i]`	`u(v)`	for each NAL HRD buffer, then each VCL one
`initial_cpb_removal_delay_offset[i]`	`u(v)`	as above

Both are initial_cpb_removal_delay_length_minus1 + 1 bits wide.

Picture timing (type 1)

Name	Coding	Condition
`cpb_removal_delay`	`u(v)`	NAL or VCL HRD present
`dpb_output_delay`	`u(v)`	NAL or VCL HRD present
`pic_struct`	`u(4)`	`pic_struct_present_flag` set; see below
`clock_timestamp_flag[i]`	`u(1)`	for each clock timestamp
`ct_type`	`u(2)`	0 progressive, 1 interlaced, 2 unknown
`nuit_field_based_flag`	`u(1)`
`counting_type`	`u(5)`	0 to 6
`full_timestamp_flag`	`u(1)`
`discontinuity_flag`	`u(1)`
`cnt_dropped_flag`	`u(1)`
`n_frames`	`u(8)`
`seconds_value`, `minutes_value`, `hours_value`	`u(6)`, `u(6)`, `u(5)`	present directly when `full_timestamp_flag` is 1, or nested behind `seconds_flag`, `minutes_flag`, `hours_flag` otherwise
`time_offset`	`i(v)`	`time_offset_length` bits

`pic_struct`	Meaning	Clock timestamps
0	frame	1
1	top field	1
2	bottom field	1
3	top then bottom field	2
4	bottom then top field	2
5	top, bottom, top repeated	3
6	bottom, top, bottom repeated	3
7	frame doubling	2
8	frame tripling	3

User data

Type 4, user data registered by ITU-T T.35, starts with a one-byte country code, then an extension byte if that byte was FF, then the registrant’s data. Type 5, user data unregistered, starts with a 16-byte UUID that names its owner, followed by the data. Encoders often use type 5 to record their name and settings.

Recovery point (type 6)

Name	Coding	Range
`recovery_frame_cnt`	`ue(v)`	0 to 65535
`exact_match_flag`	`u(1)`
`broken_link_flag`	`u(1)`
`changing_slice_group_idc`	`u(2)`	0 to 2

A decoder that starts at this message has correct output after recovery_frame_cnt more frames, which allows joining a stream without an IDR picture.

Other messages

The fields of the pan-scan, film grain, frame packing and display orientation messages, confirmed against FFmpeg’s syntax, are:

Message	Fields
Pan scan rectangle (2)	`pan_scan_rect_id`, `pan_scan_rect_cancel_flag`, `pan_scan_cnt_minus1` (0 to 2), then for each rectangle four `se(v)` offsets, `pan_scan_rect_repetition_period`
Film grain (19)	`film_grain_characteristics_cancel_flag`, `film_grain_model_id` (2 bits, 0 to 1), `separate_colour_description_present_flag` and its six colour fields, `blending_mode_id` (2 bits), `log2_scale_factor` (4 bits), `comp_model_present_flag[c]`, and for each plane `num_intensity_intervals_minus1`, `num_model_values_minus1`, lower and upper bounds and `comp_model_value`, then `film_grain_characteristics_repetition_period`
Frame packing (45)	`frame_packing_arrangement_id`, `frame_packing_arrangement_cancel_flag`, `frame_packing_arrangement_type` (7 bits), `quincunx_sampling_flag`, `content_interpretation_type` (6 bits), `spatial_flipping_flag`, `frame0_flipped_flag`, `field_views_flag`, `current_frame_is_frame0_flag`, `frame0_self_contained_flag`, `frame1_self_contained_flag`, four 4-bit grid positions when needed, a reserved byte, `frame_packing_arrangement_repetition_period`, `frame_packing_arrangement_extension_flag`
Display orientation (47)	`display_orientation_cancel_flag`, `hor_flip`, `ver_flip`, `anticlockwise_rotation` (16 bits, 2¹⁶ is a full turn), `display_orientation_repetition_period`, `display_orientation_extension_flag`

Frame packing type values are 0 checkerboard, 1 column, 2 row, 3 side-by-side, 4 top-bottom, 5 temporal, 6 no packing. Mastering display (137) and content light level (144) carry the SMPTE ST 2086 and CEA-861.3 HDR values. The payloads of the remaining messages are defined in the standard’s Annex D and were not confirmed from a public source.

.H264 File Format – SEI messages