JavaScript lexical structure

The source text is broken into input elements, from left to right, taking the longest possible match each time. They are white space, line terminators, comments and tokens. Tokens are what the grammar is built from; the rest is discarded, except that a line terminator can matter to automatic semicolon insertion (ECMA-262, ECMAScript Language: Lexical Grammar).

Goal symbols

The same characters can be different tokens depending on where they are, so the lexer is told what to expect by the parser. There are five goals. They differ only in whether a / starts a regular expression or a division, whether a } continues a template, and whether a hashbang is allowed:

Goal	Used where
`InputElementHashbangOrRegExp`	The very start of a script or module
`InputElementRegExp`	Where a regular expression is allowed and a template continuation is not
`InputElementRegExpOrTemplateTail`	Where both are allowed
`InputElementTemplateTail`	Where a template continuation is allowed but a regular expression is not
`InputElementDiv`	Everywhere else, so `/` and `/=` are division

White space

White space separates tokens and is otherwise ignored.

Code point	Name	Grammar name
U+0009	Character tabulation	`<TAB>`
U+000B	Line tabulation	`<VT>`
U+000C	Form feed	`<FF>`
U+FEFF	Zero width no-break space	`<ZWNBSP>`
Any code point in category Zs	Space, no-break space and the other Unicode space separators	`<USP>`

Other format-control characters (category Cf) are not white space. They may appear in comments, strings, templates and regular expressions.

Line terminators

Code point	Name	Grammar name
U+000A	Line feed	`<LF>`
U+000D	Carriage return	`<CR>`
U+2028	Line separator	`<LS>`
U+2029	Paragraph separator	`<PS>`

A CR followed by an LF counts as one line terminator where a line terminator sequence is meant, such as in a line continuation. A line terminator may not appear in a regular expression literal. <LF> and <CR> may not appear in a string literal, except in a line continuation. <LS> and <PS> have been allowed in string literals since ES2019, so that all JSON text is valid JavaScript.

Comments

Form	Name	Ends at	Where
`/* text */`	Multi-line comment	The first `*/`	Anywhere a token may be
`// text`	Single-line comment	The end of the line	Anywhere a token may be
`#! text`	Hashbang comment	The end of the line	Only the first two code points of the source; see File structure
`<!-- text`	HTML-like open comment	The end of the line	Scripts only, as below
`--> text`	HTML-like close comment	The end of the line	Scripts only, at the start of a line, as below

Comments do not nest.
A multi-line comment that contains a line terminator is treated as a line terminator when semicolons are inserted. Otherwise it counts as white space.
A */ cannot be inside a multi-line comment, and a comment’s text is never interpreted, so a Unicode escape in it has no effect.

HTML-like comments

Annex B defines two extra comment forms, kept so that old pages that wrapped scripts in HTML comments still work. Annex B is required in web browsers. It does not apply when parsing a Module, so in a module these are syntax errors.

SingleLineHTMLOpenComment::
    <!-- [SingleLineCommentChars]

SingleLineHTMLCloseComment::
    LineTerminatorSequence HTMLCloseComment

HTMLCloseComment::
    [WhiteSpaceSequence] [SingleLineDelimitedCommentSequence] --> [SingleLineCommentChars]

<!-- behaves like // anywhere a token could start.
--> behaves like // only at the start of a line, after optional white space and comments that stay on that line. It also works at the start of the input, and after a multi-line comment that contains a line terminator.
Elsewhere, --> is the tokens -- and >, as in while (n --> 0).

x = 1; <!-- x = 2;
--> x = 3;
console.log(x); // 1 in a script, syntax error in a module

Tokens

CommonToken::
    IdentifierName
    PrivateIdentifier
    Punctuator
    NumericLiteral
    StringLiteral
    Template

The /, /= and } tokens, the regular expression literal and the later parts of a template are defined separately, as they depend on the goal.

Identifier names

IdentifierName::
    IdentifierStart
    IdentifierName IdentifierPart
IdentifierStart::
    IdentifierStartChar
    \ UnicodeEscapeSequence
IdentifierPart::
    IdentifierPartChar
    \ UnicodeEscapeSequence
IdentifierStartChar::
    UnicodeIDStart
    $
    _
IdentifierPartChar::
    UnicodeIDContinue
    $

UnicodeIDStart and UnicodeIDContinue are the code points with the Unicode properties ID_Start and ID_Continue. _ is in ID_Continue but not ID_Start, so the grammar lists it for the start.
Names may be spelled with Unicode escapes: caf\u00e9 and café are the same name. An escape must stand for a code point that is valid in that position.
Names are case-sensitive, and no normalisation takes place.
A PrivateIdentifier is # followed by an identifier name, with no space between: #count. See Classes.

Punctuators

Kind	Tokens
Grouping and separators	`{` `}` `(` `)` `[` `]` `.` `...` `;` `,` `?.`
Comparison	`<` `>` `<=` `>=` `==` `!=` `===` `!==`
Arithmetic	`+` `-` `` `/` `%` `*` `++` `--`
Shift and bitwise	`<<` `>>` `>>>` `&` `\|` `^` `~`
Logical and conditional	`!` `&&` `\|\|` `??` `?` `:`
Assignment	`=` `+=` `-=` `=` `/=` `%=` `*=` `<<=` `>>=` `>>>=` `&=` `\|=` `^=` `&&=` `\|\|=` `??=`
Arrow	`=>`

?. is not recognised when a decimal digit follows it, so a?.5:1 is a conditional expression. Meanings and precedence are on the Expressions page.

Keywords and reserved words

ECMA-262 lists 38 reserved words (Keywords and Reserved Words):

await break case catch class const continue debugger default delete do else
enum export extends false finally for function if import in instanceof new
null return super switch this throw true try typeof var void while with yield

Every reserved word is still a valid IdentifierName, so it can be a property name: obj.class and { default: 1 } are fine. It cannot be the name of a variable, function or label. enum is reserved for future use and has no meaning yet. null, true and false are literals, but are listed here.

A keyword must be spelled literally. A code point in one cannot be a Unicode escape, and a name that would escape to a reserved word is an error, so els\u{65} cannot be used as a variable name.

Contextual words

Words	Treatment
`await`	Reserved in modules and inside async functions. An ordinary identifier elsewhere in scripts
`yield`	Reserved in strict mode code and inside generators. An ordinary identifier elsewhere in sloppy code
`let` `static` `implements` `interface` `package` `private` `protected` `public`	Reserved in strict mode code only
`as` `async` `from` `get` `meta` `of` `set` `target`	Always usable as identifiers. Keywords only in the specific places where the grammar uses them
`using`	Used as a keyword in using declarations, in the current draft (expected in ES2027). Not reserved

A few words have rules of their own. A LexicalDeclaration may not bind the name let, even in sloppy code. In strict mode code, arguments and eval may not be declared or assigned: see Strict mode.

Automatic semicolon insertion

Most statements end with a semicolon, but the source may leave some out, and the parser inserts them (Automatic Semicolon Insertion). The three rules are:

When the parser meets a token that no production allows, called the offending token, a semicolon is inserted before it if the offending token is on a later line than the previous token, or it is }, or the previous token is ) and the new semicolon would end a do-while statement.
When the end of the input is reached and the tokens do not form a whole Script or Module, a semicolon is inserted at the end.
When a token is allowed by the grammar, but follows a place marked [no LineTerminator here] in a restricted production, and a line terminator comes before it, a semicolon is inserted before that token.

One condition overrides them all: a semicolon is never inserted if it would be parsed as an empty statement, or become one of the two semicolons in a for header. So in if (a > b) at the end of the input, no semicolon is inserted, as it would be the empty body of the if.

Restricted productions

Production	Written	A line break before the marked part means
Postfix `++` and `--`	`a` ⏎ `++`	The `++` applies to the next operand, so `a` is left unchanged
`return`	`return` ⏎ `value`	`return;`
`throw`	`throw` ⏎ `value`	A syntax error, since `throw;` is not valid
`break`, `continue`	`break` ⏎ `label`	`break;` and `continue;`, with the label as the next statement
`yield`	`yield` ⏎ `value`	`yield` with no operand
Arrow functions	`(a)` ⏎ `=> a`	A syntax error
`async` before `function` or a name	`async` ⏎ `function f() {}`	`async` is an identifier expression, and the declaration follows
`using` (draft)	`using` ⏎ `x = y;`	`using` is an identifier expression, not a declaration

Where semicolons are not inserted

No semicolon is inserted when the next line can continue the expression, because then there is no offending token. A line starting with one of (, [, `, /, + or - joins the previous line.

const a = b
(c || d).run()        // parsed as  const a = b(c || d).run();

const list = get()
[1, 2].forEach(f)     // parsed as  const list = get()[1, 2].forEach(f);

const n = m
/foo/g.test(s)        // parsed as  const n = m / foo / g.test(s);

return
  1 + 2               // parsed as  return; 1 + 2;

The first three lack an inserted semicolon, and the last gains one

Literals

Kind	Examples	Value
Null	`null`	The Null type
Boolean	`true` `false`	The Boolean type
Numeric	`42` `1.5e3` `0xFF` `0o17` `0b11` `1_000`	The Number type
BigInt	`10n` `0xFFn`	The BigInt type
String	`'a'` `"b"` `'\n'`	The String type
Template	`a ${b}`	The String type, or whatever a tag returns
Regular expression	`/ab+c/gi`	A new `RegExp` object

Array and object literals are expressions rather than tokens. They are on the Expressions page.

Numeric literals

NumericLiteral::
    DecimalLiteral
    DecimalBigIntegerLiteral
    NonDecimalIntegerLiteral [BigIntLiteralSuffix]
    LegacyOctalIntegerLiteral
DecimalLiteral::
    DecimalIntegerLiteral . [DecimalDigits] [ExponentPart]
    . DecimalDigits [ExponentPart]
    DecimalIntegerLiteral [ExponentPart]
BinaryIntegerLiteral::   0b BinaryDigits  |  0B BinaryDigits
OctalIntegerLiteral::    0o OctalDigits   |  0O OctalDigits
HexIntegerLiteral::      0x HexDigits     |  0X HexDigits
LegacyOctalIntegerLiteral:: 0 OctalDigit  |  LegacyOctalIntegerLiteral OctalDigit
BigIntLiteralSuffix::    n
NumericLiteralSeparator:: _

Form	Example	Value	Since
Decimal	`255` `2.5` `.5` `5.` `1e3`	255, 2.5, 0.5, 5, 1000	ES1
Hexadecimal	`0xFF`	255	ES1
Octal	`0o377`	255	ES2015
Binary	`0b11111111`	255	ES2015
BigInt	`255n` `0xFFn`	255n	ES2020
With separators	`1_000_000`	1000000	ES2021
Legacy octal	`0377`	255, sloppy code only	Annex B

A number is an IEEE 754 double. A decimal literal is rounded to the nearest value, which may lose precision: see Types.
Separators may go between digits only, one at a time, so 1__0, _1, 1_, 0_1 and 1_.5 are invalid.
BigInt literals may not have a fraction or an exponent, and may not use the legacy octal form.
Legacy forms, allowed in sloppy code by Annex B: a leading 0 and octal digits is legacy octal, so 010 is 8. With 8 or 9 present, it is decimal, so 09 is 9. Both are syntax errors in strict mode code.
The code point right after a number may not be an identifier start or a digit, so 3in x is invalid, while 3 in x is fine.
A number followed by a property access needs care: 1.toString() is a syntax error, since the dot belongs to the number. Write 1..toString(), 1 .toString() or (1).toString().

String literals

A string is zero or more characters in single or double quotes. The value is a sequence of UTF-16 code units.

Escape	Value
`\b`	Backspace, U+0008
`\t`	Tab, U+0009
`\n`	Line feed, U+000A
`\v`	Vertical tab, U+000B
`\f`	Form feed, U+000C
`\r`	Carriage return, U+000D
`\"` `\'` `\\`	The quote or backslash itself
`\0`	NUL, U+0000, when not followed by a decimal digit
`\xHH`	The code unit with that two-digit hex value
`\uHHHH`	The code unit with that four-digit hex value
`\u{H…}`	A code point up to `10FFFF`, as one or two code units (ES2015)
`\` + line terminator sequence	Line continuation: contributes no characters
`\` + any other character	The character itself
`\1` to `\377`, `\8`, `\9`	Legacy octal and non-octal escapes, sloppy code only; syntax errors in strict mode code

Template literals

Template::
    NoSubstitutionTemplate
    TemplateHead
NoSubstitutionTemplate::   ` [TemplateCharacters] `
TemplateHead::             ` [TemplateCharacters] ${
TemplateMiddle::           } [TemplateCharacters] ${
TemplateTail::             } [TemplateCharacters] `

A template is written in backticks and may span lines. CR and CRLF in the source become LF in the value.
${ starts a substitution, ending at the matching }. Any expression may go inside, including another template.
The same escapes work as in strings, except that legacy octal forms are never allowed. A \ followed by 0 and a decimal digit is an error in an untagged template.

const who = 'world';
const greeting = `Hello, ${who}!
Sum: ${1 + 2}`;
// "Hello, world!\nSum: 3"

A tagged template calls a function with the literal parts and the values. The first argument is an array of the cooked strings, with a raw property holding the text as written. The array and its raw are frozen, and the same object is passed every time a given template site is evaluated.

function tag(strings, ...values) {
  return strings.raw.join('|') + ' ' + values.join(',');
}
tag`a${1}\n${2}b`;        // "a|\\n|b 1,2"
String.raw`C:\new`;       // "C:\\new"

String.raw is a built-in tag that returns the raw text

Since ES2018, a tagged template may contain invalid escapes such as \unicode. The cooked string for that part is undefined, and the raw string holds the text.

Regular expression or division

RegularExpressionLiteral::
    / RegularExpressionBody / RegularExpressionFlags
RegularExpressionFirstChar::
    RegularExpressionNonTerminator but not one of * or \ or / or [
    RegularExpressionBackslashSequence
    RegularExpressionClass
RegularExpressionFlags::
    [empty]
    RegularExpressionFlags IdentifierPartChar

The lexer cannot tell a regular expression from a division on its own: the parser chooses the goal from what it expects next. A / where the grammar expects an operator is division, and one where it expects an expression is a regular expression.

a / b / c;                // two divisions
const re = /a/g;          // a regular expression
x = a++ / 2;              // division
if (ok) /x/.test(s);      // a regular expression: the ")" ends the if head
f(a) / 2;                 // division: the ")" ends a call

After ) the choice depends on what the ) closed, so a tokeniser alone cannot decide

The body cannot start with *, because /* starts a comment, and cannot be empty, because // starts a comment.
A / inside a class needs no backslash: /[/]/ is valid.
No line terminator is allowed in the literal.
The pattern’s syntax and flags are checked when the code is parsed, so a bad literal is an early error. Each evaluation creates a new object.

The pattern language is on the Regular expressions page.

.JS File Format – Lexical structure