RKT file format: Macros and syntax objects

A macro is a function from syntax to syntax that the compiler runs before the program does. Racket's macro system is the most developed of the Lisp family: macros are hygienic (a name a macro introduces cannot clash with the user's), they work across modules, and they can build whole languages. The data a macro handles is not a bare S-expression but a syntax object.

Syntax objects

A syntax object (syntax?) wraps a datum with the information the expander needs: the source location (syntax-source, syntax-line, syntax-column, syntax-position, syntax-span), the lexical context that says what each identifier means, and syntax properties (syntax-property), for example the 'paren-shape that records which brackets the reader saw. An identifier is a syntax object that wraps a symbol (identifier?).

Function	Does
`syntax-e`	takes the outermost layer off: the symbol, a list of syntax objects, a vector, and so on
`syntax->datum`, `datum->syntax`	strip a syntax object to a plain datum; wrap a datum, taking its lexical context from another syntax object. `(datum->syntax ctx 'name)` makes an identifier that is not hygienic, to bind a name the user's code can see
`syntax->list`	the elements of a syntax list as a list of syntax objects
`free-identifier=?`, `bound-identifier=?`	whether two identifiers refer to the same binding; whether one would capture the other
`generate-temporaries`, `format-id` (`racket/syntax`)	fresh identifiers; an identifier made from a format string
`raise-syntax-error`	a syntax error that names the form and the piece of syntax at fault

Defining macros

Form	Use
`(define-syntax-rule (name pattern …) template)`	a macro with one pattern and one template. `(define-syntax-rule (swap! a b) (let ([t a]) (set! a b) (set! b t)))`
`(define-syntax name (syntax-rules (literal …) [(_ pattern …) template] …))`	several patterns. A name in the literals list matches only that word; `_` in a pattern matches anything
`(define-syntax (name stx) body …)`	a macro written as a procedure from a syntax object to a syntax object, run at compile time. It can use any code, such as `syntax-case` or `syntax-parse`, and report errors
`(define-syntax name (lambda (stx) …))`, `(define-syntaxes (a b) …)`	the long forms; `define-syntax` binds a name to a transformer, a procedure that the expander calls on a use of the name
`(let-syntax …)`, `(letrec-syntax …)`, `(syntax-id-rules …)`	macros bound locally; a macro that also controls how its bare name and `set!` are used
`(define-simple-macro (name . pattern) body)`, `(define-syntax-parse-rule (name . pattern) body)`	`syntax/parse/define`: `define-syntax-rule` with `syntax-parse` patterns, so a pattern variable can carry a class such as `x:id`
`(make-rename-transformer id)`, `(make-set!-transformer proc)`, a structure with `prop:rename-transformer`	make a name stand for another, or make a variable-like macro

#lang racket
(define-syntax-rule (while test body ...)
  (let loop () (when test body ... (loop))))

(define i 0)
(while (< i 3) (displayln i) (set! i (add1 i)))

(define-syntax my-or
  (syntax-rules ()
    [(_) #f]
    [(_ e) e]
    [(_ e rest ...) (let ([t e]) (if t t (my-or rest ...)))]))   ; t is renamed, so no capture

Two pattern macros. The macro's own t cannot capture a variable named t in the user's expression.

Patterns and templates

The pattern language of syntax-rules, define-syntax-rule and syntax-case is the same:

Pattern	Matches
`_`	any syntax, binding nothing
an identifier (not a literal and not `...`)	any syntax; the identifier is a pattern variable
a literal identifier	an identifier that has the same binding
`(p …)`, `(p …+ . np)`	a list, or an improper list with a tail pattern
`(p … p ellipsis p … . np)`	a list in which one pattern may repeat; `...` is the ellipsis, and the variables under it bind to sequences
`#(p …)`, `#&p`, `#s(key p …)`	a vector, a box, a prefab structure
`(... ...)` and a constant	a literal ellipsis; a datum that is `equal?` (a number, string, boolean and so on)

(syntax-case stx-expr (literal …) [pattern result-expr] [pattern fender result-expr] …) finds the first matching pattern whose optional fender expression is true and evaluates the result, which uses the pattern variables only through syntax. (syntax template) or #'template builds a syntax object: each pattern variable is replaced by what it matched, x ... repeats the element for each match, (... ...) writes a literal ..., (~? a b) chooses the first template if its variables are all bound and the second otherwise, and (~@ . t) splices into the list around it. quasisyntax (#`) with #,expr and #,@expr evaluates expressions inside a template; syntax/loc gives the result the source location of another piece of syntax; with-syntax binds pattern variables in a body.

(require (for-syntax racket/base))           ; compile-time code needs the base language

(define-syntax (my-struct stx)
  (syntax-case stx ()
    [(_ name (field ...))
     (with-syntax ([make (datum->syntax #'name
                           (string->symbol (format "make-~a" (syntax-e #'name))))])
       #'(struct name (field ...) #:constructor-name make))]))

(define-syntax (unless2 stx)
  (syntax-case stx ()
    [(_ test body ...)
     (identifier? #'test)                         ; a fender
     #'(if test (void) (begin body ...))]))

A syntax-case macro with with-syntax and datum->syntax, and one with a fender.

syntax-parse

syntax/parse gives a richer matcher with good error messages. (syntax-parse stx-expr option … [pattern directive … body] …) uses syntax classes on pattern variables: x:id (an identifier), e:expr, n:number, s:str, k:keyword, and user-defined ones from define-syntax-class and define-splicing-syntax-class. Its patterns add ~or, ~and, ~not, ~seq, ~optional, ~var, ~datum, ~literal, ~describe, ~bind, ~fail, ~parse, ~rest, ~between, ~once and the cut ~!. Options include #:literals, #:datum-literals and #:context, and a clause may use #:with, #:when and #:attr directives between the pattern and the body.

(require (for-syntax racket/base syntax/parse))

(define-syntax (my-let stx)
  (syntax-parse stx
    [(_ ([x:id e:expr] ...) body:expr ...+)
     #'((lambda (x ...) body ...) e ...)]))

A syntax-parse macro: a binding that is not an identifier is rejected with a precise message.

Hygiene, scopes and phases

Hygiene. The expander marks each piece of syntax with scopes. A binding introduced by a macro gets the macro's scope and so cannot be seen by code the user wrote, and a free name in the template refers to what the macro's module saw. datum->syntax with the user's syntax as the context is how a macro deliberately breaks this.
Phases. A macro runs at phase 1, the compile time of the module that uses it. The libraries that a macro's code uses must be imported for-syntax, and definitions it needs go in (begin-for-syntax …) or define-for-syntax. This is why a module that defines macros typically has (require (for-syntax racket/base)).
Expansion. The expander reads a form, finds its head, and if the head is bound to a macro calls it and expands the result again; local-expand and syntax-local-expand-expression let a macro ask for the expansion of a subform. expand and expand-once show what a form becomes.
Syntax parameters (racket/stxparam): (define-syntax-parameter it (λ (stx) (raise-syntax-error …))) and (syntax-parameterize ([it …]) body) make a name that means something different inside a macro's use, as this does in a class.
Lifting. syntax-local-lift-expression, syntax-local-lift-module, syntax-local-lift-require and syntax-local-lift-provide move a definition, a submodule, a require or a provide out of the form that the macro is expanding.

Macros that make languages

Every form that Racket reads is wrapped by #%app (a procedure call), #%datum (a literal), #%top (an unbound name), #%module-begin (the module body) and #%top-interaction (the REPL). Because they are bindings, a module language can export its own versions: a language can make every number a string, every call a trace, or every module print its last value. The #%module-begin a language gives is what runs on the whole body, which is how languages such as typed/racket and lazy change a module. syntax/module-reader makes the matching reader, and (provide (all-from-out racket/base) (rename-out [my-app #%app])) is the typical export. See The #lang family.

Not described

Known, not yet covered

Reference: syntax/parse (its syntax-class forms, ~ patterns, attributes, #:with, define-eh-alternative-set and error reporting) is described from memory of the Racket manuals: that library's documentation is in a separate package and was not downloaded. Treat that section as a sketch.
Reference: the sets-of-scopes model of binding, the phases and binding spaces in detail, syntax-local-* and local-expand arguments, taint and arm (syntax-arm, syntax-disarm, syntax-protect), certificates in older versions, racket/stxparam and the macro-writing libraries (syntax/stx, syntax/srcloc, racket/syntax) were not read.
Viewer: the patterns of syntax-parse are coloured as patterns, with the ~ words as keywords and x:class names as variables, but the #:with pattern after a clause is read as code.

.RKT File Format – Macros and syntax objects