.C File Format – Lexical structure

Tokens and white space

token:
    keyword  |  identifier  |  constant  |  string-literal  |  punctuator

preprocessing-token:
    header-name  |  identifier  |  pp-number  |  character-constant
    string-literal  |  punctuator
    each non-white-space character that cannot be one of the above

White space is the space, horizontal tab, new-line, vertical tab and form feed, and comments, which count as white space. Between tokens it has no meaning, except that a new-line ends a preprocessing directive.

The tokeniser always takes the longest sequence that can form a token, which is called the maximal munch rule (C17 6.4). This explains some famous errors:

x+++y         /* read as x ++ + y, not x + ++ y */
x+++++y       /* read as x ++ ++ + y: an error, although x++ + ++y is valid */
a/*b          /* the start of a comment, not a / *b */
0xE+1         /* one pp-number, which is an invalid token: write 0xE + 1 */

The last example comes from the preprocessing number token, which covers more text than any real constant (C17 6.4.8). It starts with a digit, or a dot and a digit, then continues with digits, letters, underscores, dots, and e+, e-, E+, E-, p+, p-, P+ and P-. So 0xE+1 is one pp-number, which is not a valid constant.

Comments

Comments do not nest, and neither form means anything inside the other or inside a string or character constant. Each comment is replaced by one space in phase 3 (C17 6.4.9), so a/**/b is two tokens. Because this happens before preprocessing, comments may appear inside directives and a /* */ comment may continue past the end of a directive’s line.

The // form relies on line splicing having run first. A backslash at the end of a // comment continues the comment onto the next line.

Before C99, a C compiler could read a//* comment */ b as a / b. This is one of the very few changes where a valid C90 program changes meaning in C99.

To disable a block of code containing comments, use #if 0 rather than another comment.

Keywords

Keywords are reserved and cannot be identifiers (C17 6.4.1). The list has grown with each standard:

The underscore-and-capital spelling was used for new keywords because it comes from the reserved part of the identifier space, so existing programs could not clash with them. Headers offered lower-case macros: <stdbool.h> defined bool, for example. C23 made the common ones real keywords and keeps the old spellings as alternatives:

_Noreturn is also deprecated in C23, in favour of the [[noreturn]] attribute. asm and fortran are common extensions, not C keywords.

Identifiers

identifier:
    identifier-nondigit
    identifier identifier-nondigit
    identifier digit

identifier-nondigit:
    nondigit  |  universal-character-name  |  other implementation-defined characters

• An identifier begins with a Latin letter or _ and continues with letters, digits and _. Identifiers are case-sensitive.
• From C99, a universal character name, \uXXXX or \UXXXXXXXX, may name a letter from another script, within the ranges given in Annex D. Compilers may also accept these characters written directly. In C23, the permitted characters are defined by Unicode Standard Annex 31.
• In C17, a universal character name may not name a character below U+00A0, other than $, @ and `, or one in the range U+D800 to U+DFFF (C17 6.4.3).
• Compilers often accept $ in identifiers as an extension.
• Only the first 63 characters of an internal identifier and the first 31 of an external one are guaranteed to be significant. C90 guaranteed only 31 and 6 (see Translation limits).
• __func__ is a predefined identifier, not a macro or a keyword (see Functions).

Integer constants

integer-constant:
    decimal-constant [integer-suffix]
    octal-constant [integer-suffix]
    hexadecimal-constant [integer-suffix]
    binary-constant [integer-suffix]                     (C23)

decimal-constant:      nonzero-digit {digit}
octal-constant:        0 {octal-digit}
hexadecimal-constant:  0x hex-digits  |  0X hex-digits
binary-constant:       0b binary-digits  |  0B binary-digits   (C23)

integer-suffix:  u  l  ll  and combinations, in either case
                 wb  uwb  in either case                       (C23)

A leading 0 means octal: 010 is 8, and 09 is an error. 0 itself is an octal constant. In C23 a single ' may separate digits: 1'000'000 or 0xFFFF'0000.

There are no negative integer constants. -1 is the unary minus operator applied to the constant 1. This is why INT_MIN cannot be written as -2147483648 on a 32-bit int: 2147483648 does not fit and gets a wider type.

Floating constants

floating-constant:
    decimal-floating-constant  |  hexadecimal-floating-constant

decimal-floating-constant:
    fractional-part [exponent-part] [suffix]
    digit-sequence  exponent-part   [suffix]

exponent-part:           e [sign] digits  |  E [sign] digits
hexadecimal-floating-constant:    (C99)
    0x hex-fraction  p [sign] digits  [suffix]
    0x hex-digits    p [sign] digits  [suffix]
suffix:                  f  l  F  L
                         df dd dl DF DD DL   (C23, decimal floating types)

The type is double without a suffix, float with f or F, and long double with l or L. A decimal constant needs a dot, an exponent, or both, so 1., .5, 1e3 and 1.5e-3 are all floating constants, and 1 is not.

A hexadecimal constant (C99) requires the binary exponent, written p, which is a power of two given in decimal. 0x1.8p1 is 1.5 × 2, or 3.0. Hexadecimal constants let a value be written exactly, without decimal rounding. C23 digit separators work in floating constants as they do in integers.

Character constants

A plain character constant has type int in C, not char as in C++: sizeof 'a' is sizeof(int). It may be negative if char is signed: with a signed 8-bit char, '\xFF' is −1.

A constant holding more than one character, such as 'ab', has an implementation-defined value. C23 removed multi-character u and U constants. A u8 constant must fit a single UTF-8 code unit, so only characters U+0000 to U+007F may be used.

String literals

• A string literal has static storage duration. Its array has one more element than the characters, for the terminating null.
• Its type is an array of non-const char, but modifying it is undefined behaviour (C17 6.4.5).
• Adjacent literals are joined in phase 6. A literal with no prefix can join one with a prefix and takes that prefix. Joining different prefixes is implementation-defined in C11 and not allowed in C23.
• Equal literals may or may not occupy the same storage. This is unspecified.

char a[] = "text";      /* array of 5 char, may be changed */
const char *p = "text"; /* pointer to a string literal: do not modify it */
char b[4] = "text";     /* valid in C: 4 elements, no null terminator */
char c[3] = "text";     /* an error: too many characters */

Escape sequences

These are allowed in character constants and string literals (C17 6.4.4.4). An unrecognised escape is not defined by the standard.

A hexadecimal escape takes every hexadecimal digit that follows, so "\x41BC" is one character with value 0x41BC, not an A then BC. Split the string to stop it: "\x41" "BC". The value of a hexadecimal or octal escape must fit the character type (C17 6.4.4.4). \e for escape is a common extension, not part of the standard.

Trigraphs

Trigraphs let C be written on keyboards and character sets that lacked some of its symbols. They are replaced in phase 1, before anything else (C17 5.2.1.1). This happens even inside string literals, which is why \? exists. C23 removed them.

GCC leaves trigraphs alone unless -trigraphs or a strict -std= option is given. Because ??/ becomes a backslash in phase 1, it can even join two lines of a comment.

Digraphs

Digraphs arrived in C95, again for keyboards without the punctuators. Unlike trigraphs they are real tokens, so they are not replaced inside strings or comments (C17 6.4.6).

The header <iso646.h> gives macro names for more operators: and, or, not, and_eq, or_eq, xor, xor_eq, not_eq, bitand, bitor and compl.

Punctuators

A punctuator is a symbol with independent syntactic meaning (C17 6.4.6). Each operator’s meaning is on the Expressions page.

In C23 attributes use [[ and ]], written as pairs of bracket tokens, and :: joins an attribute’s prefix to its name.

Header names

A header name, <name> or "name", is a token only in #include directives, the C23 #embed directive and __has_include and __has_embed expressions. A name may not contain a new-line, and using ', \, // or /* in it is undefined behaviour (C17 6.4.7).

Other constants

• An enumeration constant is an identifier declared in an enum. Before C23 it has type int.
• true and false are keywords in C23, with type bool. In earlier standards they are macros from <stdbool.h> expanding to 1 and 0.
• nullptr (C23) is a predefined constant of type nullptr_t (see Null pointers).