Emacs has its own regular expression engine, a backtracking matcher with a syntax that is close to POSIX basic expressions with some extensions. A regexp is a string, so every backslash of the regexp must be doubled inside the Lisp string: the regexp \(a\|b\)+ is written "\\(a\\|b\\)+". In the tables below the regexp column shows the regexp itself; double each backslash when you put it in a string. The same text can also be built with the rx notation, which avoids the doubling. A syntax table tells the matcher and the motion commands what each character means in a given language.
The regexp language
| Regexp | Matches |
|---|---|
| ordinary character | itself; all except . * + ? [ ] ^ $ \ (and the combination *? and so on) are ordinary |
. | any character except a newline |
*, +, ? | zero or more, one or more, zero or one of the previous item; greedy |
*?, +?, ?? | the same, but non-greedy (as few as possible) |
\{m\}, \{m,\}, \{,n\}, \{m,n\} | an interval: exactly m, at least m, at most n, or between m and n repeats |
[abc], [a-z] | a character set; a range uses -; ] first in the set is a literal, - first or last is a literal, and a backslash is literal inside a set |
[^abc] | a complemented set; it matches a newline too |
^, $ | start and end of a line; special only at the start or end of the regexp or of a group or alternative, and literal elsewhere |
\ followed by a special character | quotes that character |
\| | alternation; the longest alternative is not preferred, the first that leads to a match wins |
\( … \) | a group that is numbered from 1 by the position of its opening parenthesis |
\(?: … \) | a shy group: groups without numbering |
\(?N: … \) | an explicitly numbered group |
\N (digit 1 to 9) | a back-reference: the same text as group N matched |
\w, \W | a word-constituent character; a character that is not one |
\sC, \SC | a character of syntax class C (\s- whitespace, \s_ symbol); a character that is not |
\cC, \CC | a character that is in category C (for example \cg for Greek); that is not |
\`, \' | start and end of the buffer or string (the accessible part, for a buffer) |
\= | the position of point (buffers only) |
\b, \B | at a word boundary, not at one |
\<, \> | start and end of a word |
\_<, \_> | start and end of a symbol |
Character classes
A class is written inside a set: [[:alpha:][:digit:]_]. It cannot stand alone.
| Class | Matches |
|---|---|
[:alpha:], [:alnum:] | letters (Unicode-aware); letters and digits |
[:digit:], [:xdigit:] | the digits 0 to 9; hexadecimal digits |
[:space:], [:blank:] | characters with whitespace syntax in the current syntax table; horizontal white space (space, tab and other Unicode blanks) |
[:upper:], [:lower:] | upper-case and lower-case letters; with case-fold-search on, each matches any letter |
[:punct:] | ASCII punctuation, and in other scripts any non-word-syntax character |
[:word:] | characters with word syntax |
[:cntrl:], [:graph:], [:print:] | control characters; printable characters except the space; printable characters including the space |
[:ascii:], [:nonascii:] | codes 0 to 127; all others |
[:multibyte:], [:unibyte:] | multibyte characters; unibyte characters (including raw bytes) |
Backtracking
The matcher tries alternatives and repetitions in order and goes back when a later part fails. A greedy .* first takes everything it can and then gives characters back; the non-greedy .*? takes as few as possible. Nested repetitions such as \(a*\)*b can take exponential time on a failing input. Emacs 30 added re-disassemble, which shows the compiled form of a regexp, to look inside the matcher.
(string-match "\\([0-9]+\\)-\\([0-9]+\\)" "tel 555-1234") ; => 4
(match-string 2 "tel 555-1234") ; => "1234"
(string-match "\\_<if\\_>" "(if x y)") ; => 1
(string-match "\\(.\\)\\1" "hello") ; => 2 (the "ll")Building regexps
| Function | Meaning |
|---|---|
regexp-quote | escapes special characters so that a string matches itself |
regexp-opt | an efficient regexp for a list of strings; the optional paren argument t wraps the result in a capturing group, and words or symbols add word or symbol boundaries |
regexp-opt-depth | the number of capturing groups of a regexp |
regexp-opt-charset | a set expression for a list of characters |
Searching and matching
| Function | Meaning |
|---|---|
search-forward, search-backward | search for fixed text from point; arguments are the text, a bound, noerror and a repeat count; point moves to the end (or start) of the match |
re-search-forward, re-search-backward | the same with a regexp |
looking-at, looking-at-p | whether the text at point matches (the second keeps the match data) |
looking-back | whether the text before point matches; the limit argument bounds the search, and it can be slow |
string-match, string-match-p | search a string, optionally from a start index; return the position |
match-beginning, match-end, match-string, match-data, set-match-data | the positions and text of the last match and its groups; group 0 is the whole match, an unused group gives nil |
replace-match | replaces the last match in the buffer or in a string; in the new text \& is the whole match, \N a group and \\ a backslash, unless the literal argument is set |
replace-regexp-in-string | returns a copy of a string with every match replaced by text or by a function’s result |
save-match-data | restores the match data after the body |
The variable case-fold-search (default t) makes a search ignore case, and a search string that contains upper-case letters is still case-insensitive in Lisp (only the interactive commands change it). search-spaces-regexp replaces each run of spaces in a regexp by another regexp for commands that use it. text-property-search-forward finds text by property (Buffers and text). Incremental search (isearch) is a command with its own variables and is only named here.
(with-temp-buffer
(insert "foo=1, bar=22")
(goto-char (point-min))
(let (pairs)
(while (re-search-forward "\\([a-z]+\\)=\\([0-9]+\\)" nil t)
(push (cons (match-string 1) (string-to-number (match-string 2))) pairs))
(nreverse pairs))) ; => (("foo" . 1) ("bar" . 22))The rx notation
(rx FORM…) is a macro that builds a regexp string at compile time from S-expressions; rx-to-string does the same at run time. It chooses the shy groups and the brackets for you.
| Form | Regexp |
|---|---|
| a string, a character | the text, quoted |
(seq …), (: …), (and …), (sequence …) | concatenation; with no argument it matches the empty string (Emacs 27) |
(or …), (| …) | alternation; with no argument it never matches (Emacs 27); unmatchable is the same |
(any "a-z" ?_), (in …), (char …) | a set of characters, ranges and classes, e.g. (any digit "a-f") |
(not (any …)), (not-char …), (not digit) | a complemented set or class |
(literal EXPR), (regexp EXPR) | insert a string expression, quoted or as a regexp; EXPR may be any Lisp, which makes rx build the string at run time (Emacs 27) |
bol, eol, bos, eos (bot, eot) | ^, $, \`, \' |
point, word-boundary, not-word-boundary, bow, eow | \=, \b, \B, \<, \> |
symbol-start, symbol-end | \_<, \_> |
digit, alpha, alnum, space, blank, upper, lower, punct, word, xdigit, cntrl, graph, print, ascii, nonascii | the character classes above (some have long aliases such as numeric, alphabetic, whitespace) |
nonl (not-newline), anychar (anything) | .; any character including a newline (anychar since Emacs 27) |
(group …), (submatch …), (group-n N …) | a numbered group; a group with an explicit number |
(backref N) | \N |
(* …), (+ …), (? …), (zero-or-more …), (one-or-more …), (opt …) | repetition, greedy |
(*? …), (+? …), (?? …) | non-greedy repetition |
(= N …), (>= N …), (** N M …), (repeat N M …) | exactly N; at least N; between N and M |
(minimal-match …), (maximal-match …) | make *, + and ? inside non-greedy or greedy |
(syntax CLASS), (category CAT) | \sC with a class name such as whitespace or punctuation; \cC |
(eval EXPR) | evaluates EXPR at expansion time; from Emacs 31 it uses the current Lisp dialect instead of always dynamic binding |
(rx bol (group (+ (any "a-z"))) "=" (group (+ digit)) eol)
;; => "^\\([a-z]+\\)=\\([[:digit:]]+\\)$"New forms are added with (rx-define NAME [ARGS] RX) globally, (rx-let ((NAME [ARGS] RX)…) BODY) locally in a body, and rx-let-eval for run-time use (all Emacs 27). Emacs 27 also made reversed ranges in any an error. In Emacs 31 the atom any (an old alias for nonl, often used by mistake for anychar) is obsolete and warns; the list form (any …) is not affected. Emacs 31 also requires the correct spelling of the category chinese-two-byte.
Syntax tables
Every buffer has a syntax table, a char-table that gives each character a syntax descriptor. It decides what a word is for forward-word and \w, how parentheses and strings nest for forward-sexp, and where comments are. \s in a regexp and the skip functions use the same classes.
| Char | Class | Meaning |
|---|---|---|
- or space | whitespace | white space |
w | word | part of a word |
_ | symbol | part of a symbol, not a word |
. | punctuation | separator |
( | open parenthesis | opens a list; the matching character follows in the descriptor |
) | close parenthesis | closes a list |
' | expression prefix | a quote or other prefix of an expression |
" | string quote | starts and ends a string with the same character |
$ | paired delimiter | delimits like a string but is not a quote (TeX’s dollar) |
\ | escape | quotes the next character, in strings and outside |
/ | character quote | quotes the next character only inside strings |
<, > | comment start, comment end | delimiters of a comment (a newline usually ends it) |
@ | inherit | use the standard syntax table’s entry |
! | generic comment fence | a character that starts and ends a comment, set with the syntax-table property |
| | generic string fence | the same for strings |
Flags
Characters after the class and the matching character add flags. 1 and 2: the character is the first or second of a two-character comment starter. 3 and 4: the first or second of a two-character comment ender. p: a prefix character. b: the comment style is b (instead of a). n: the comment can nest. c: the comment style is c. C’s /* and */ comments are described by ". 14" for / and ". 23" for *.
Raw descriptors and functions
The descriptor stored in the table is a cons (CODE . MATCHING-CHAR). CODE is the class number (0 whitespace, 1 punctuation, 2 word, 3 symbol, 4 open, 5 close, 6 prefix, 7 string, 8 paired, 9 escape, 10 character quote, 11 comment start, 12 comment end, 13 inherit, 14 comment fence, 15 string fence) plus the flags as bits starting at 1<<16 for flag 1. string-to-syntax converts a descriptor string to this form.
| Function | Meaning |
|---|---|
modify-syntax-entry | sets the entry of a character or a (FROM . TO) range, e.g. (modify-syntax-entry ?_ "w" table) |
char-syntax, syntax-class-to-char, syntax-after, syntax-class | read the class of a character as a character, from a class number, or at a position |
standard-syntax-table, syntax-table, set-syntax-table, make-syntax-table, copy-syntax-table | access and create tables; make-syntax-table inherits from the standard table |
with-syntax-table | runs the body with another table current |
parse-sexp-ignore-comments | when non-nil, expression motion skips comments |
syntax-ppss, parse-partial-sexp | parse from a start position; syntax-ppss caches the state, so it is the one to call |
syntax-propertize-function | a function that sets syntax-table text properties on a region, for syntax that depends on context (here-documents, regexp literals) |
The parser state
| Index | Content |
|---|---|
| 0 | depth in parentheses |
| 1 | start of the innermost containing list |
| 2 | start of the last complete sexp |
| 3 | the character that ends the string, t for a generic fence, or nil outside a string |
| 4 | nil outside a comment, t in a non-nestable one, or the nesting depth |
| 5 | t if the position follows a quote character |
| 6 | the minimum depth reached during the parse |
| 7 | comment style: nil for a, 1 for b, 2 for c, or syntax-table for a generic comment |
| 8 | start of the comment or string, or nil |
| 9 | list of the positions of the open parentheses, outermost first |
| 10 | when the parse stopped inside a two-character construct, the position of its first character, otherwise nil |
(with-temp-buffer
(emacs-lisp-mode)
(insert "(a \"b")
(let ((s (syntax-ppss)))
(list (nth 0 s) (nth 3 s)))) ; => (1 34)Since Emacs 29 the variable syntax-wholeline-max limits the cost of long lines in parsing. Since Emacs 31, characters whose Unicode general category is Nd have the category digit (mnemonic 6) and characters of the symbol script have the category symbol (mnemonic 5).
Not described
Known, not yet covered
- Reference: the list of character categories (
\cCletters),define-category, and the category tables. - Reference: case tables (
standard-case-table,set-case-syntax-pair) and their effect on matching. - Reference: the regexp cache and the limits of the matcher (
regexp-unmatchable, the stack limit,re-searcherrors such asinvalid-regexp), and the match-data functions with thereseatargument. - Reference: the rest of the
rxforms and aliases (for exampleline-start,string-start,not-wordchar,rx-let-evaldetails) and the Perl-compatible regexps in other libraries (pcre,re-builder). - Reference: syntax-table text properties in detail and
syntax-propertize-rules.
See also
- Strings and characters – backslashes in Lisp strings
- Buffers and text – point, motion and text properties
- Lexical structure – how the reader treats the characters of the source
- Lists and hash tables – char-tables
- Searching and Matching – the manual’s chapter on regexps and search
- Syntax Tables – the manual’s chapter on syntax tables
- The rx Structured Regexp Notation – the manual’s section on rx