.HTML File Format – Encoding and media type

Choosing the encoding (13.2.3)

Before tokenizing, a browser settles on a character encoding:

1. A byte order mark at the start: EF BB BF is UTF-8, FE FF UTF-16 big-endian, FF FE UTF-16 little-endian. It overrides everything else.
2. The transport: a charset parameter on the HTTP Content-Type header. (A file opened from disk has none.)
3. The prescan: the first 1024 bytes are scanned for a <meta> tag (below). The value is looked up in the Encoding Standard’s list of labels. A label of UTF-16 is treated as UTF-8, because the prescan would not have worked on real UTF-16; x-user-defined becomes Windows-1252.
4. Otherwise a guess, typically Windows-1252 for Western locales, or UTF-8 if the bytes are valid UTF-8.

Conforming documents use UTF-8 and declare it with <meta charset="utf-8">, fully inside the first 1024 bytes.

The prescan

The prescan walks the first 1024 bytes, skipping comments (<!-- … -->) and any other tags (<x …>, <!…>, <?…>, and their attributes, read carefully so that a > in a quoted value does not end the tag early). The first <meta tag with a usable charset decides:

• <meta charset="x"> gives x.
• <meta http-equiv="Content-Type" content="text/html; charset=x"> gives x, but only if the http-equiv attribute is present.

A <meta> inside a comment therefore does not count, which is the viewer’s reading too.

Media type and extensions

XHTML, the XML form of HTML, uses application/xhtml+xml and an XML parser; a file served as text/html is parsed as HTML whatever it looks like.