Common Lisp has no lexer separate from its parser. The reader turns characters into Lisp objects directly, one object at a time, using a table (the readtable) that says what each character does. This page describes the reader of the standard readtable (CLHS chapter 2); a program can change the readtable, so no description of the text of a Lisp file is final until the code that precedes it has run.
Character syntax types
Every character has one of these syntax types in the standard readtable.
| Type | Characters | Effect |
|---|---|---|
| Whitespace | Space, Tab, Newline, Linefeed, Return, Page (form feed), Rubout | Separates tokens; otherwise skipped |
| Terminating macro | " ' ( ) , ; ` | Calls a reader function, and ends a token that is being read |
| Non-terminating macro | # | Calls a reader function at the start of a token; inside a token it is an ordinary character |
| Single escape | \ | The next character is part of a token, with no special meaning and with its case kept |
| Multiple escape | | | Everything up to the next unescaped | is part of the token, with its case kept |
| Constituent | letters, digits, and ! $ % & * + - . / : < = > ? @ [ ] ^ _ { } ~ and Backspace | Part of a token |
Characters outside the standard set (accented letters, Greek, and so on) are constituents in the standard readtable, if the implementation supports them. The characters [ ] { } are reserved for the programmer: the standard keeps them constituents so that libraries can make them macro characters, and some do (cl-interpol, named-readtables based syntaxes, and so on). The viewer treats them as ordinary constituents.
The reader algorithm
- Read a character x. If it is whitespace, discard it and repeat. A character that is invalid in the standard syntax (there are none in the standard readtable apart from Backspace as a token start) signals a reader error.
- If x is a macro character, call its function. It may return an object (a list, a string, a quoted form) or return nothing at all (a comment); in the second case start again at step 1.
- If x is a single escape, read the next character and begin a token with it as an escaped constituent. If x is a multiple escape, begin a token and read to the matching multiple escape.
- Otherwise x is a constituent: begin a token with it.
- Continue the token: a constituent is appended, a single escape appends the next character literally, a multiple escape appends everything up to the next multiple escape. Whitespace or a terminating macro character ends the token (the character is put back, or discarded if whitespace and
readhas been told to consume it). - Convert the unescaped letters of the token by the readtable case (upcase by default) and interpret the token as a number, if it fits the syntax of a number, or else as a symbol.
An escaped character is never part of a number, so 1\2 is a symbol. A token made only of dots (., ..) is an error, except that a single . is the dot of a dotted list or pair when read inside a list.
Because # is non-terminating, a#b is one symbol, while a'b is the symbol a then 'b.
Case
The readtable case has four values, set with (setf (readtable-case rt) …): :upcase (the default; foo reads as FOO), :downcase, :preserve (nothing changes) and :invert (a token in all one case is inverted, a mixed-case token is kept). Escaped characters are never converted. *print-case* chooses how symbols are written back (:upcase, :downcase, :capitalize). Case is never significant in the names of characters (#\space and #\SPACE are the same), or in number syntax (1E3 and 1e3).
Comments
;to the end of the line. Any number of semicolons works; conventions use one to four.#| … |#a block comment. Blocks nest, so#| a #| b |# c |#is one comment. An unmatched#|runs to the end of the file. The text inside is not read, so unbalanced quotes and parentheses are harmless there.#+feature formand#-feature formread the form and discard it when the feature test is false: this is conditional text, described under Reader macros.- A string whose value is not used (a documentation string) is data, not a comment, though the viewer colours the usual positions like comments.
Numbers
A token is a number when it fits one of these shapes; *read-base* (default 10) is the radix of integers and ratios.
| Kind | Syntax | Examples |
|---|---|---|
| Integer | optional sign, digits, optional trailing decimal point (the point forces base 10) | 42 -7 +0 10. |
| Ratio | optional sign, digits, /, digits; always in lowest terms after reading; denominator not zero | 1/2 -22/7 4/2 (reads as 2) |
| Float | optional sign, then digits with a decimal point, and optionally an exponent; or digits with an exponent and no point | 1.5 .5 -3.14e2 1e10 1.5d0 2s0 |
| Complex | not a token: #C(real imaginary); see Reader macros | #C(1 2) #C(1.5 -0.5) |
| Radix integer or ratio | #b, #o, #x, or #nr then the digits | #b101 #o17 #xFF #36rZZ #x-1F #b11/101 |
An exponent marker is one letter and chooses the float format: s short-float, f single-float, d double-float, l long-float, and e meaning the format in *read-default-float-format* (initially single-float). A float with no marker, such as 1.5, also uses that format. The standard allows short and long floats to be the same as single and double; SBCL, CCL and most others make s a single-float and l a double-float. 1.5d0 is the portable way to write a double-float. The letters are case-insensitive.
Tokens that look nearly like numbers but are not are symbols: 1+, 1-, +, -, 1e, .e5, 1/2/3, 12abc. The standard calls tokens that could be extended number syntax by an implementation potential numbers and reserves them, but real implementations read them as symbols.
The special variables that change reading are *read-base*, *read-default-float-format*, *read-eval*, *read-suppress* and *readtable*. The viewer reads numbers in base 10 only, because *read-base* is set by running code.
Characters
#\x is the character x. A name follows when more than one character comes after #\ before the next terminator: #\Space, #\Newline, #\Tab, #\Page, #\Rubout, #\Linefeed, #\Return, #\Backspace. These eight are the standard names (Newline and Space belong to the standard characters; the others are semi-standard: the implementation need not provide them, but if it does, it uses these names). A single character is always itself, even a terminator: #\(, #\), #\;, #\", #\ (a space) and #\#. Names are case-insensitive. Other characters are written by implementation-defined names: #\U+00E9 or #\UE9 (hex code), #\LATIN_SMALL_LETTER_E_WITH_ACUTE, #\Nul, #\Escape, #\Bell and so on; char-name and name-char convert. Modifiers (#\Control-A, #\Meta-a, #\Hyper-, #\Super-) were a Lisp Machine feature mentioned in CLtL; ANSI leaves them to the implementation.
Strings
A string starts and ends with ". Inside, a backslash makes the next character literal: it is only needed before " and before \ itself. There are no other escapes in standard Lisp: "a\nb" is the string anb, not a line break (a real line break inside the quotes is a newline in the string). A string may span lines. Format directives live in strings, as Format describes. Strings read from source are literal objects and a program may not modify them. Libraries such as cl-interpol add string syntaxes with escapes by changing the readtable; none of that is standard.
Symbols
Any token that is not a number is a symbol. A symbol can be written with package markers (pkg:name, pkg::name, :name); see Symbols and packages. Escapes let a symbol hold any character: |Hello World| and Hello\ World are the same symbol, with the name Hello World; |a|b is the symbol aB (the first part escaped and kept, the rest converted to upper case); || is the symbol with the empty name. A symbol name is printed with escapes when it needs them (*print-escape*).
How the viewer colours a file
The viewer reads a file the way the reader does: it never starts a token inside a string, a comment or an escape, and it ends tokens at the terminating macro characters. Then it colours what each token is.
| Colour class | What it marks |
|---|---|
| Comment | ; lines, #| |# blocks (nested), a #! first line, and documentation strings |
| String | strings with their escapes, character literals (#\Space), the text between format directives |
| Number | integers, ratios, floats with every exponent marker, #b #o #x #nr numbers and #* bit vectors |
| Keyword (purple) | the special operators and the standard macros when they head a list, and library definers such as defsystem and any def… or with-… macro name |
| Function (blue) | any other name at the head of a list, and #'name |
| Type | standard types and condition types where a type is expected (declarations, the, typecase clauses, quoted type specifiers, specialisers, :type), and package prefixes |
| Literal (orange) | :keywords, t, nil, pi, the standard constants such as most-positive-fixnum, +constant+ names, #:uninterned symbols and the names in #+ feature expressions |
| Variable (green) | *special* variables, including the standard ones (*package*, **, ++ …) |
| Directive (magenta) | lambda-list keywords (&optional …), declaration words (optimize, speed, ignore …), loop keywords, format directives inside format strings, and the reader marks ' ` , ,@ #' #+ #- #. #c #p #s #a #1= #1# |
A name is only coloured as a call when it is where a call can be. A name in a quoted list is data and stays plain; so does the name of a variable in a let binding, a lambda list, a do or dolist header, a slot specification or a case key. After a comma inside a backquote the reader is back in code, and so is the viewer. See Functions and variables for the lists of standard names it knows.