A Prolog text is read in two steps. The tokenizer cuts the characters into tokens (ISO/IEC 13211-1, section 6.4); the reader builds terms from the tokens and ends a term at an end token. This page is the tokenizer. Every character of a file belongs to exactly one token or to layout.
Token kinds
| Kind | Written as | Notes |
|---|---|---|
| layout | spaces, tabs, line ends and other white space | separates tokens; ISO names the space, the horizontal tab and the new line. SWI-Prolog 11.0.0 takes the Unicode Pattern_White_Space characters (tab to carriage return, space, U+0085, U+200E, U+200F, U+2028, U+2029); the no-break space, U+2000 to U+200A, U+3000 and other control characters are illegal characters there, and so in the viewer |
| line comment | % to the end of the line | layout for the reader |
| block comment | /* to the first */ | ISO systems end it at the first */; SWI-Prolog 11.0.0 nests (/* a /* b */ c */ is one comment, and an inner /* without its own */ leaves the comment unclosed to the end of the file; checked with read_term/3). The viewer nests when the comment balances and otherwise stops at the first */ with a note. A comment starts only where a token can start (inside a run of symbol characters such as =/* the /* belongs to the atom) |
| name (atom) | a lower-case letter, then letters, digits and _ | foo, a_b2; a letter with no case (for example in Chinese) also starts an atom |
| symbol atom | a run of the characters #$&*+-./:<=>?@^~\ | :-, =.., \+; the longest run is one atom |
| solo atom | ! or ; | each is a token on its own |
| quoted atom | '...' | any characters, with escapes; '' inside is one quote |
| variable | an upper-case letter or _, then letters, digits and _ | X, _Rest; a lone _ is the anonymous variable |
| integer or float | see below | a minus sign before a number is a separate token that the reader joins |
| double-quoted text | "..." | read as a string, a code list, a character list or an atom, by the double_quotes flag |
| back-quoted text | `...` | a list of character codes (ISO); SWI-Prolog and others |
| punctuation | ( ) [ ] { } , | | a ( that follows a name with no layout in between is the open-ct: it makes a compound term; || is read as one atom |
| end | . followed by layout, % or the end of the file | ends a clause; any other . belongs to a symbol atom |
Numbers
| Form | Example | Notes |
|---|---|---|
| decimal integer | 0, 42 | no limit on size in most systems (unbounded integers), 64 bits in some |
| character code | 0'a, 0'\n, 0''', 0' | 0' and one character or escape; a quote is written as two quotes (this site also reads 0'') |
| binary, octal, hexadecimal | 0b1010, 0o17, 0xFF | ISO |
| radix | 2'1010, 16'FF | base 2 to 36 (SWI-Prolog, SICStus and others, not ISO) |
| float | 1.5, 1.5e10, 1.0E-3 | ISO needs digits on both sides of the point; SWI-Prolog, SICStus and YAP also read 1e10 |
| infinity, not a number | 1.0Inf, 1.5NaN | SWI-Prolog; the flags float_overflow, float_zero_div and float_undefined decide whether such values are produced or raise an error |
| digit groups | 1 000 000, 1_000_000 | SWI-Prolog 11.0.0: a single space or an underscore between two runs of digits joins them (1 2 reads as 12, and a group is not limited to three digits); a number with groups has no fraction or exponent (1 000.5 is 1000, the functional dot and 5); 0x1_F, 0b1_0 and 16'F_F take underscores too |
| rational | 1r3 | SWI-Prolog with GMP |
numbers([0'a, 0'\n, 0b1010, 0o17, 0xFF, 2'1010, 16'FF, 1.5e10, 1_000_000, 1.0Inf, 1r3]).
Quoted items and escapes
A quoted atom, a double-quoted text and a back-quoted text are written the same way; only the quote character differs. Inside, the quote character is doubled to stand for itself, and a backslash starts an escape sequence. A backslash at the end of a line continues the item on the next line and adds nothing. ISO does not allow a bare line break in a quoted item; many systems do, and this site notes it.
| Escape | Meaning | Standard |
|---|---|---|
\a \b \f \n \r \t \v | alert, backspace, form feed, new line, return, tab, vertical tab | ISO |
\\ \' \" \` | the character itself | ISO |
\0 to \7, then more octal digits and \ | the character with that octal code (\101\) | ISO (the closing backslash is optional in SWI-Prolog) |
\xHH..\ | the character with that hexadecimal code | ISO (the closing backslash is optional in SWI-Prolog) |
\uXXXX, \UXXXXXXXX | a Unicode code point of four or eight hexadecimal digits | SWI-Prolog and others |
\e, \s | escape (27), space | SWI-Prolog and others |
\c | skips the layout that follows | SWI-Prolog |
| backslash and a line end | continues the item on the next line | ISO |
Any other escape is a syntax error. The viewer lists it as an error, with its line.
message('It''s a quote, a tab\t and a line break\n').
text("two lines \
joined").
codes(`abc`).
unicode('\x41\\u00e9').
How the viewer colours the tokens
| Colour | Tokens |
|---|---|
| comment | % and /* */ comments |
| string | double-quoted and back-quoted text (a double-quoted text after set_prolog_flag(double_quotes, atom) is coloured as an atom) |
| literal | quoted atoms and [] |
| number | every number form, and a minus sign that is part of a negative number |
| variable | variables, including _ |
| function | the name at the head of a clause |
| keyword | built-in predicates and control: !, ;, ->, \+, is, mod, true, write(...) and the like |
| directive | :-, ?-, -->, => and the name after a leading :- |
| type | the tag of a SWI-Prolog dict (point{x: 1}) |
Not described
Known, not yet covered
- Viewer: the character set of a text (the
encodingflag and the:- encoding(...)directive) is not followed: a file is read as UTF-8, or as Windows-1252 when it is not valid UTF-8. - Viewer: the character-conversion table (
char_conversion/2) and thevar_prefixflag of SWI-Prolog change what a name means and are not followed. - Viewer: a bare line break inside a quoted item is accepted and noted, but a system that rejects it is not told apart from one that accepts it.
- Reference: SWI-Prolog 11.0.0 reads a non-ASCII character as follows (found by reading every character up to U+2FFFF with
read_term/3): an upper-case letter (Lu) starts a variable; ID_Start letters start an atom; marks and digits (ID_Continue) continue a name but cannot start one; every other symbol or punctuation character is an atom of one character and is never joined to its neighbours (so≤=is two tokens); opening brackets (Ps), controls, private-use and unassigned characters are illegal; the quotation marks « ‘ “ ‹ start a text that ends at the matching mark and is read as a string. A short list of exceptions is in the source of the viewer's tokenizer. A quoted atom is an operator in SWI-Prolog only when its text could not be written without quotes ('my op', not'-'or'mod'); ISO says every quoted atom is. The other systems' sets are not listed, and the numbers that other systems read (such as0'with a character in Cor.2 form) are covered by their common forms only.
Comments, layout and the end token
A
%comment runs to the end of the line. A/* ... */comment may span lines (it nests in SWI-Prolog, see the table); a comment that is not closed runs to the end of the file, which the viewer lists as an error. Comments are layout:foo(a, /* b */ c)reads likefoo(a, c).The end token is a full stop followed by layout, a
%or the end of the file. A full stop insideX = a.bor=..is part of a symbol atom, andX = 'end.'is a quoted atom. The last clause of a file must end the same way; a file that stops without it has a problem, which the viewer lists.