C++ lexical structure

After line splicing (see File structure), the compiler divides the text into preprocessing tokens, and later into tokens. There are five kinds of token: identifiers, keywords, literals, operators and punctuators [lex.token].

Tokens

White space (spaces, tabs, vertical tabs, form feeds, new-lines) and comments separate tokens and are otherwise ignored after preprocessing.

preprocessing-token:
    header-name
    import-keyword
    module-keyword
    export-keyword
    identifier
    pp-number
    character-literal
    user-defined-character-literal
    string-literal
    user-defined-string-literal
    preprocessing-op-or-punc
    each non-white-space character that cannot be one of the above

The tokeniser always takes the longest sequence that could form a token, a rule known as maximal munch [lex.pptoken]. So x+++++y reads as x ++ ++ + y, which does not compile, although x++ + ++y would. There are three exceptions:

<:: is not read as the digraph <: followed by : unless the next character is : or >. This lets std::vector<::Foo> compile.
After #include and in certain import directives, <name> is one header-name token.
In a template argument list, the first >> that is not nested closes two lists [temp.names], so vector<vector<int>> works (C++11).

Comments

Form	Ends at
`/* text */`	The first `*/`; may span lines
`// text`	The end of the line, which a final backslash can extend

Comments do not nest. Inside either kind, the markers for the other have no meaning. Each comment becomes one space in phase 3 [lex.comment]. Documentation comments such as /// and /** */ are a tool convention, not part of the language.

Identifiers

An identifier starts with a letter or underscore, and continues with letters, digits and underscores. Since C++23 the permitted letters are defined by Unicode: a start character is XID_Start or _, and later characters are XID_Continue. The identifier must be in Normalization Form C [lex.name]. Identifiers are case-sensitive and have no length limit in the language.

Some identifiers are reserved for the implementation and must not be used in a program [lex.name]:

Any identifier that contains a double underscore __.
Any identifier that starts with an underscore and a capital letter.
Any identifier in the global namespace that starts with an underscore.
The names of standard library entities, in the places the library reserves.

Four identifiers have a special meaning in certain places and are ordinary names elsewhere:

Identifier	Special where	Since
`override`	After a member function declarator	C++11
`final`	After a member function declarator or a class name	C++11
`import`	Starting an import declaration	C++20
`module`	Starting a module declaration	C++20

Keywords

C++23 has 81 keywords, which can never be identifiers [lex.key]. The table shows when each was added. C++14, C++17 and C++23 added none.

Added in	Count	Keywords
C++98	63	`asm` `auto` `bool` `break` `case` `catch` `char` `class` `const` `const_cast` `continue` `default` `delete` `do` `double` `dynamic_cast` `else` `enum` `explicit` `export` `extern` `false` `float` `for` `friend` `goto` `if` `inline` `int` `long` `mutable` `namespace` `new` `operator` `private` `protected` `public` `register` `reinterpret_cast` `return` `short` `signed` `sizeof` `static` `static_cast` `struct` `switch` `template` `this` `throw` `true` `try` `typedef` `typeid` `typename` `union` `unsigned` `using` `virtual` `void` `volatile` `wchar_t` `while`
C++11	10	`alignas` `alignof` `char16_t` `char32_t` `constexpr` `decltype` `noexcept` `nullptr` `static_assert` `thread_local`
C++20	8	`char8_t` `concept` `consteval` `constinit` `co_await` `co_return` `co_yield` `requires`

Some keywords have changed meaning:

Keyword	Change
`auto`	A storage class in C++98. Since C++11 a placeholder for a deduced type
`export`	Marked exported templates in C++98. That meaning was removed in C++11, and C++20 reused the word for modules
`register`	A storage class until C++17, when it lost its meaning. It stays reserved
`typename`	Still a keyword, but C++20 no longer requires it in many contexts where only a type can appear

true, false and nullptr are keywords that are also literals.

Alternative tokens

Some punctuators have an alternative spelling [lex.digraph]. Each one behaves exactly like its primary form, apart from its spelling when converted to a string by the # operator. The named forms are built into the language, not macros, so a header such as <ciso646> was never needed.

Alternative	Primary	Alternative	Primary
`<%`	`{`	`and`	`&&`
`%>`	`}`	`and_eq`	`&=`
`<:`	`[`	`bitand`	`&`
`:>`	`]`	`bitor`	`\|`
`%:`	`#`	`compl`	`~`
`%:%:`	`##`	`not`	`!`
		`not_eq`	`!=`
		`or`	`\|\|`
		`or_eq`	`\|=`
		`xor`	`^`
		`xor_eq`	`^=`

Before C++17 there were also nine trigraphs, such as ??= for #. C++17 removed them.

Operators and punctuators

The list below is every operator and punctuator [lex.operators]. Their meanings are on the Expressions page.

Group	Tokens
Brackets and separators	`{` `}` `[` `]` `(` `)` `;` `:` `...` `?` `,`
Scope and member access	`::` `.` `.` `->` `->`
Arithmetic	`+` `-` `*` `/` `%` `++` `--`
Bitwise	`~` `&` `\|` `^` `<<` `>>`
Logical	`!` `&&` `\|\|`
Comparison	`==` `!=` `<` `>` `<=` `>=` `<=>`
Assignment	`=` `+=` `-=` `*=` `/=` `%=` `^=` `&=` `\|=` `<<=` `>>=`
Preprocessor only	`#` `##` (and `%:` `%:%:`)

Literals

Kind	Examples	Type
Integer	`42` `0x2A` `052` `0b101010` `42u` `42LL` `42uz`	Depends on value and suffix
Floating-point	`1.5` `.5` `1e10` `2.5f` `1.5L` `0x1.8p1`	`double` unless suffixed
Character	`'a'` `L'a'` `u8'a'` `u'a'` `U'a'`	`char`, `wchar_t`, `char8_t`, `char16_t`, `char32_t`
String	`"hi"` `L"hi"` `u8"hi"` `u"hi"` `U"hi"`	Array of the matching `const` character type
Raw string	`R"(a\b)"`	As for the string prefix
Boolean	`true` `false`	`bool` [lex.bool]
Pointer	`nullptr`	`std::nullptr_t` [lex.nullptr]
User-defined	`12_km` `"x"s` `10ms`	The return type of the literal operator

Integer literals

integer-literal:
    binary-literal [integer-suffix]
    octal-literal [integer-suffix]
    decimal-literal [integer-suffix]
    hexadecimal-literal [integer-suffix]

binary-literal:       0b binary-digit | 0B binary-digit | binary-literal ['] binary-digit
octal-literal:        0 | octal-literal ['] octal-digit
decimal-literal:      nonzero-digit | decimal-literal ['] digit
hexadecimal-literal:  hexadecimal-prefix hexadecimal-digit-sequence

Base	Prefix	Example	Value	Since
Decimal	none, first digit not 0	`255`	255	C++98
Hexadecimal	`0x` or `0X`	`0xFF`	255	C++98
Octal	`0`	`0377`	255	C++98
Binary	`0b` or `0B`	`0b1111'1111`	255	C++14

The type of an integer literal is the first in the list that can hold its value [lex.icon]. A decimal literal without u never becomes unsigned; other bases can.

Suffix	Decimal	Binary, octal or hexadecimal
none	`int`, `long`, `long long`	`int`, `unsigned int`, `long`, `unsigned long`, `long long`, `unsigned long long`
`u`	`unsigned int`, `unsigned long`, `unsigned long long`
`l`	`long`, `long long`	`long`, `unsigned long`, `long long`, `unsigned long long`
`ul`	`unsigned long`, `unsigned long long`
`ll`	`long long`	`long long`, `unsigned long long`
`ull`	`unsigned long long`
`z` (C++23)	The signed type matching `std::size_t`	The signed type matching `std::size_t`, then `std::size_t`
`uz` (C++23)	`std::size_t`

Each suffix letter may be upper or lower case, but ll must match: LL or ll, never Ll. The u may come before or after the other part.
There are no negative literals. -1 is the unary minus applied to the literal 1.
A leading zero makes the literal octal: 010 is 8. 09 is an error.
A literal too large for all the listed types is ill-formed, unless an extended integer type can hold it.

Digit separators

Since C++14 a single quote may appear between any two digits of a numeric literal. It has no effect on the value. The grammar allows one quote between digits, and not at the start or end of the digits, or next to a prefix or suffix.

long budget   = 1'000'000;
int  mask     = 0b1010'0101;
int  rgb      = 0xFF'80'00;
double pi     = 3.141'592'653;
int  octal    = 0'777;        // valid: a single 0 followed by ' and octal digits

// 1''000    two quotes in a row: invalid
// 0x'FF     next to the prefix: invalid
// 100'      at the end: invalid

The quote may divide any digits; the groups need not be of equal size

Floating-point literals

decimal-floating-point-literal:
    fractional-constant [exponent-part] [floating-point-suffix]
    digit-sequence exponent-part [floating-point-suffix]

hexadecimal-floating-point-literal:
    hexadecimal-prefix hexadecimal-fractional-constant binary-exponent-part [floating-point-suffix]
    hexadecimal-prefix hexadecimal-digit-sequence binary-exponent-part [floating-point-suffix]

exponent-part:          e [sign] digit-sequence | E [sign] digit-sequence
binary-exponent-part:   p [sign] digit-sequence | P [sign] digit-sequence

Suffix	Type	Since
none	`double`	C++98
`f` `F`	`float`	C++98
`l` `L`	`long double`	C++98
`f16` `F16`	`std::float16_t`	C++23
`f32` `F32`	`std::float32_t`	C++23
`f64` `F64`	`std::float64_t`	C++23
`f128` `F128`	`std::float128_t`	C++23
`bf16` `BF16`	`std::bfloat16_t`	C++23

Hexadecimal floating-point literals (C++17) give an exact binary value. The exponent after p is a decimal power of two and is required, so 0x1.8p1 is 1.5 × 2, or 3.0. A value outside the range of its type is ill-formed [lex.fcon].

Character literals

Form	Type	Since
`'a'`	`char`	C++98
`L'a'`	`wchar_t`	C++98
`u8'a'`	`char8_t` (`char` before C++20)	C++17
`u'a'`	`char16_t`	C++11
`U'a'`	`char32_t`	C++11

Unlike C, an ordinary character literal has type char, not int, so sizeof('a') is 1.
A multicharacter literal such as 'abcd' has type int and an implementation-defined value. Support is conditional.
A literal with a prefix must hold a single code unit of its encoding. Since C++23 an ordinary or wide literal that cannot be encoded in one code unit is ill-formed.

Escape sequences

Escape	Meaning
`\n`	New-line
`\t`	Horizontal tab
`\v`	Vertical tab
`\b`	Backspace
`\r`	Carriage return
`\f`	Form feed
`\a`	Alert
`\\`	Backslash
`\?`	Question mark
`\'`	Single quote
`\"`	Double quote
`\0` to `\777`	Octal escape: one to three octal digits
`\xhh`	Hexadecimal escape: one or more hexadecimal digits, with no upper limit on the count
`\o{...}`, `\x{...}`	Delimited octal and hexadecimal escapes (C++23)

A hexadecimal escape stops only at the first non-hexadecimal character, so "\x41BC" is one escape with the value 0x41BC. The delimited forms avoid this.

Universal character names

Form	Example	Since
`\uXXXX`	`\u00E9`	C++98
`\UXXXXXXXX`	`\U0001F600`	C++98
`\u{X...}`	`\u{E9}`	C++23
`\N{NAME}`	`\N{LATIN SMALL LETTER E WITH ACUTE}`	C++23

They can appear in identifiers, literals and elsewhere. Each must name a valid Unicode scalar value [lex.charset].

String literals

Form	Type of each element	Encoding	Since
`"text"`	`const char`	Ordinary literal encoding	C++98
`L"text"`	`const wchar_t`	Wide literal encoding	C++98
`u8"text"`	`const char8_t` (`const char` before C++20)	UTF-8	C++11
`u"text"`	`const char16_t`	UTF-16	C++11
`U"text"`	`const char32_t`	UTF-32	C++11

A string literal is an array, with a terminating null character added. The type of "abc" is const char[4].
It has static storage duration. Whether two equal literals share storage is unspecified, and writing to one is undefined behaviour.
Converting a string literal to char* (without const) has been ill-formed since C++11.
Adjacent literals are joined in phase 6: "foo" "bar" is "foobar". If one has an encoding prefix and the other none, the result has the prefix. Combining different prefixes is ill-formed.

Raw strings

A raw string literal (C++11) has no escape sequences, and may span lines. It takes an R after any encoding prefix [lex.string].

raw-string:
    " [d-char-sequence] ( [r-char-sequence] ) [d-char-sequence] "

d-char:  any member of the translation character set except
         space, ( ) \ and the control characters for tab, vertical tab, form feed and new-line

The delimiter, at most 16 characters, is optional. It lets the text contain )", because only the closing sequence with the same delimiter ends the literal.

auto path  = R"(C:\Users\name\file.txt)";
auto regex = R"(\d+\.\d*)";
auto block = R"sql(SELECT name
FROM users
WHERE note = ")";)sql";     // the delimiter "sql" allows )" in the text
auto wide  = LR"(wide raw string)";

A raw string keeps every character, including new-lines and backslashes

User-defined literals

A user-defined literal (C++11) is a literal followed by an identifier suffix. The compiler calls a literal operator to produce its value [lex.ext], [over.literal]. A suffix must start with an underscore. Suffixes without one are reserved for the standard library.

Literal	Operator form
Integer	`operator""_s(unsigned long long)`, or the raw form `operator""_s(const char*)`, or `template<char...> operator""_s()`
Floating-point	`operator""_s(long double)`, or the raw or template form above
Character	`operator""_s(char)`, or with `wchar_t`, `char8_t`, `char16_t` or `char32_t`
String	`operator""_s(const char*, std::size_t)`, the same with other character types, or a literal operator template with one class-type non-type template parameter (C++20)

constexpr long double operator""_km(long double value)
{
    return value * 1000.0L;
}

constexpr auto distance = 2.5_km;   // 2500.0L

Standard suffix	Meaning	Inline namespace in `std::literals`
`s` (on a string)	`std::basic_string`, C++14	`string_literals`
`sv`	`std::basic_string_view`, C++17	`string_view_literals`
`h` `min` `s` `ms` `us` `ns`	`std::chrono::duration`, C++14	`chrono_literals`
`y` `d`	`std::chrono::year` and `day`, C++20	`chrono_literals`
`i` `if` `il`	`std::complex`, C++14	`complex_literals`

These are declared in inline namespaces, so using namespace std::literals; brings all of them into scope.

.CPP File Format – Lexical structure