A scene file is text. POV-Ray reads it in three steps. The scanner (scanner.cpp) turns bytes into lexemes: a word, a number, a string, or a single operator. The raw tokenizer (rawtokenizer.cpp) gives each lexeme a token number: a reserved word, a float, a string, an operator, or an identifier. The parser (Get_Token in parser_tokenizer.cpp) then looks identifiers up in its symbol tables and acts on # directives. This page describes the first two steps and the start of the third: what the text of a file may contain, and how it is cut into tokens. The words themselves are on Reserved words, the directives on Preprocessor directives, and the grammar of expressions on Expressions, vectors and identifiers.
Token kinds
Every lexeme becomes exactly one of these tokens. Comments and white space are not tokens; they only separate them.
| Kind | Token | Written as | Notes |
|---|---|---|---|
| byte order mark | UTF8_SIGNATURE_TOKEN | the bytes EF BB BF at the very start of a file | only recognised as the first thing in a file; it switches the file to UTF-8 |
| reserved word | the token of the word (SPHERE_TOKEN and so on) | one of the 523 words | case sensitive; see Reserved words |
| identifier | IDENTIFIER_TOKEN (and later a typed identifier token) | a word that is not reserved | looked up in the symbol tables by the parser |
| float literal | FLOAT_TOKEN | 12, 1.5, .5, 5., 1e-3 | never has a sign; see Numbers |
| string literal | STRING_LITERAL_TOKEN | "text" | may run over several lines; see Strings |
| operator or punctuation | one of 34 tokens (PLUS_TOKEN, LEFT_CURLY_TOKEN and so on) | one printable ASCII character, or >=, <=, != | see Operators and punctuation |
| directive marker | HASH_TOKEN | # | is a token in the operator table, but the parser handles it before any other code sees it; see The # character |
| end of file | END_OF_FILE_TOKEN | the end of the input | an include file ends at its end; the main file ends the scene |
After the scanner, the parser turns an identifier token into a typed token when the name is declared (FLOAT_ID_TOKEN, OBJECT_ID_TOKEN and the others listed on Reserved words). That step is symbol lookup, not scanning, and is described on the expressions page.
Characters and encoding
Outside strings and comments, a scene file may only contain printable ASCII characters (0x20 to 0x7E) and the white space characters below. Inside strings and comments any character is allowed.
| Input | Behaviour |
|---|---|
| byte order mark EF BB BF at the start | read as UTF8_SIGNATURE_TOKEN; the file is then decoded as UTF-8 and any byte sequence that is not valid UTF-8 is an error |
| no byte order mark | the encoding is detected: the first byte above 0x7F decides. If the bytes there are valid UTF-8, the rest of the file is read as UTF-8; if not, the rest of the file is read as Windows-1252. A file that mixes both is therefore an error as soon as an invalid UTF-8 sequence follows a valid one |
| a non-ASCII character outside a string or comment | error: Illegal character c5 in input file. (the hexadecimal code point, or U+xxxx above 0xFF) |
| an ASCII control character other than white space, or 0x7F | error: Illegal character 01 in input file. |
| invalid UTF-8 | error: Invalid byte sequence in UTF-8 input file (detail).; the details are lead octet expected, continuation octet expected, overlong encoding, invalid UCS code point and surrogate code point |
The scanner has decoders for ASCII, Latin-1 and Mac OS Roman as well, but the parser only ever switches to UTF-8 (on the byte order mark). The old charset setting of global_settings does not change how the file is scanned; in 3.8 it only chooses the character set that text objects use for old scenes, and it gives a warning. Each file, including each include file, is detected on its own.
A character in a string literal is stored as a 16-bit code (UCS-2). A character above U+FFFF is cut to its low 16 bits (see Strings).
White space and line ends
White space separates tokens and is otherwise ignored. Exactly five characters count as white space:
| Code | Name |
|---|---|
| 0x09 | tab |
| 0x0A | line feed |
| 0x0D | carriage return |
| 0x1A | the DOS end-of-file character (Ctrl-Z) |
| 0x20 | space |
Form feed (0x0C), vertical tab (0x0B) and the non-breaking space are not white space and are errors outside strings and comments. A line ends at LF or CR. The first line-end character that the scanner meets sets the line-end style of the file (LF for Unix files, CR for Windows files, whose LF is then skipped without counting a second line). Line and column numbers in messages follow that style. Inside a string literal, the line end of that style becomes a single LF in the string; the other character is dropped.
Identifiers and reserved words
An identifier is one ASCII letter or _, followed by any number of ASCII letters, digits or _. The scanner takes the longest such run, so box2 is one word, and 2box is a number followed by a word. There is no limit on the length. Letters outside ASCII are not allowed. Case matters: Name_1 and name_1 are two names, and Box is not the reserved word box.
The raw tokenizer looks the whole word up in the table of reserved words. A word that is in the table becomes the token of that word. Any other word becomes an identifier: it gets a number above the highest reserved token, and the same word always gets the same number. Two words are special even though they are reserved: local and global are read as identifiers when they are not part of a directive, because local.name and global.name reach the local and the global dictionary.
When a reserved word is tested with #ifdef or #ifndef the parser warns: Trying to test whether a reserved keyword is defined. Test result may not be what you expect.
Numbers
Every number in a scene is a float: there is no separate integer type and no hexadecimal, octal or binary notation. A float literal has this form, read by the scanner without looking at the value:
float = digits [ "." [ digits ] ] [ exponent ]
| "." digits [ exponent ]
exponent = ( "e" | "E" ) [ "+" | "-" ] [ digits ]
digits = one or more of 0 to 9| Text | Read as | Note |
|---|---|---|
12 | 12 | |
1.5 | 1.5 | |
.5 | 0.5 | a . followed by a digit starts a number |
5. | 5 | the fraction may be empty |
5.e1 | 50 | an empty fraction may be followed by an exponent |
1e+2, 1E-2 | 100, 0.01 | |
007.50 | 7.5 | leading zeros are allowed |
2e | 2 | the scanner accepts an exponent with no digits (also 2e+); the value is then taken from the text by sscanf, which on this system reads the 2 and ignores the rest. Where the C library reports a failure for such text, the parser stops with Invalid numeric literal '2e'. |
1.2.3 | 1.2 and .3 | two numbers; the second starts at the second dot |
0x10 | 0, then the word x10 | there is no hexadecimal notation; the word is an identifier, so the line is a parse error |
2x | 2, then x | a number ends at the first character that cannot continue it |
. not followed by a digit | the dot operator | member access, V.x |
A number has no sign. -3 is the operator - followed by the number 3, and - 3 is the same. The value of a literal is converted by sscanf with the %lf format into a double, so the usual limits of a double apply.
#declare A = .5;
#declare B = 5.;
#declare C = 2e;
#declare D = 1e+2;
#declare E = 5.e1;
#declare V = <1.2.3>;
#debug concat(str(A,0,1), " ", str(B,0,1), " ", str(C,0,1), " ", str(D,0,1), " ", str(E,0,1), " ", vstr(2,V,",",0,1), "\n")0.5 5.0 2.0 100.0 50.0 1.2,0.3. The vector <1.2.3> has two components, 1.2 and .3, with no comma between them.Strings
A string literal starts and ends with a double quote. It may contain any character except an unescaped ", and it may run over several lines. A backslash starts an escape sequence and the character after it is never the end of the string, so \" does not end it. A string that is not closed before the end of the file is an error: Unterminated string literal in input file. There is no single-quote string.
| Escape | Value |
|---|---|
\" | a double quote |
\' | a single quote |
\\ | a backslash |
\a | bell, 0x07 |
\b | backspace, 0x08 |
\t | tab, 0x09 |
\n | line feed, 0x0A |
\v | vertical tab, 0x0B |
\f | form feed, 0x0C |
\r | carriage return, 0x0D |
\uXXXX | the character with that code point; exactly four hexadecimal digits |
\UXXXXXX | the character with that code point; exactly six hexadecimal digits |
A code point in \u or \U must be a Unicode scalar value: not above 0x10FFFF and not a surrogate (0xD800 to 0xDFFF). Any other character after the backslash, a \u with fewer than four digits or a bad code point is an invalid escape sequence. Strings are stored as 16-bit characters, so a code point above 0xFFFF is cut to its low 16 bits (asc("\U01F600") is 62976, which is 0xF600). A character written directly in the file is decoded as UTF-8 (or Windows-1252, see above) in the same way, so "é" and "\u00e9" are the same one-character string, and strlen("é€") is 2.
Ambiguous strings and file names
A string that contains a backslash other than \" is called ambiguous, because older versions did not read escape sequences in file names and a Windows path such as "c:\temp" would have a backslash in it. The scanner therefore keeps two values for it: the string with the escapes applied, and the text as written. Which one is used depends on where the string is used and on the language version of the scene.
| Where the string is used | Result |
|---|---|
as an ordinary string (concat, #debug, text objects and so on) | the escapes are applied; an invalid escape sequence is the error Illegal escape sequence '\q' in string literal., reported when the string is used, not when it is read |
as a file name (#include, #fopen, image files, #version 3.8 or later) | the escapes are applied as for any string, so "inc\colors.inc" fails with Illegal escape sequence '\c' in string literal.; where the system uses backslash as path separator the parser also warns about it. Use a forward slash as path separator |
as a file name when the scene says #version below 3.8 | the text is used as written, with the backslashes in it, and the parser warns: Backslash encountered while parsing for a filename. In legacy (pre-v3.8) scenes, this is NOT interpreted as the start of an escape sequence. |
An invalid escape sequence in a string that is never used as a value (for instance in a skipped #if branch) causes no error.
#declare S = "tab\there \"quoted\" \\ back \u00e9 end";
#debug concat("length ", str(strlen(S), 0, 0), "\n")
#declare M = "one
two";
#debug concat("multi-line length ", str(strlen(M), 0, 0), "\n")length 30 and multi-line length 7. The first string holds 30 characters once the escapes are applied, and the second holds the line end as one LF character (one, LF, two).When #debug and the other message directives print a character above 0x7F, the text may not survive: in the check above, "caf\u00e9" printed as caf followed by a space.
Operators and punctuation
Any other printable ASCII character is a single-character token, except that !, < and > join a following = to make !=, <= and >=. These 34 tokens are the operator entries of reservedwords.cpp. The scanner does not know what they mean; the parser decides. The last column says what the parser does with each of them.
| Text | Token | Used for |
|---|---|---|
& | AMPERSAND_TOKEN | logical AND in an expression |
@ | AT_TOKEN | nothing; the parser never accepts it |
` | BACK_QUOTE_TOKEN | nothing |
\ | BACK_SLASH_TOKEN | nothing outside a string (inside a string it starts an escape) |
| | BAR_TOKEN | logical OR in an expression |
: | COLON_TOKEN | the second half of ? :; separates a name or key from its value in dictionary entries (.name: value, ["key"]: value) and the term from its coefficient in xyz(i, j, k): c |
, | COMMA_TOKEN | separates items; most commas are optional |
- | DASH_TOKEN | subtraction and negation |
$ | DOLLAR_TOKEN | nothing |
= | EQUALS_TOKEN | assignment in #declare, #local and #for; comparison for equality in an expression |
! | EXCLAMATION_TOKEN | logical NOT in an expression |
# | HASH_TOKEN | starts a directive; see The # character |
^ | HAT_TOKEN | nothing |
< | LEFT_ANGLE_TOKEN | opens a vector; less-than in an expression |
{ | LEFT_CURLY_TOKEN | opens a block |
( | LEFT_PAREN_TOKEN | opens an argument list or groups an expression |
[ | LEFT_SQUARE_TOKEN | array index, map entry, array initialiser |
% | PERCENT_TOKEN | nothing (the remainder is the word mod) |
. | PERIOD_TOKEN | member access (V.x, C.red, Dict.name) |
+ | PLUS_TOKEN | addition and unary plus |
? | QUESTION_TOKEN | the conditional operator c ? a : b |
>= | REL_GE_TOKEN | greater than or equal |
<= | REL_LE_TOKEN | less than or equal |
!= | REL_NE_TOKEN | not equal |
> | RIGHT_ANGLE_TOKEN | closes a vector; greater-than in an expression |
} | RIGHT_CURLY_TOKEN | closes a block |
) | RIGHT_PAREN_TOKEN | closes ( |
] | RIGHT_SQUARE_TOKEN | closes [ |
; | SEMI_COLON_TOKEN | ends a #declare or #local of a float, vector or colour, and a #version; from language version 3.5 on the parser expects it there and reports its absence |
' | SINGLE_QUOTE_TOKEN | nothing |
/ | SLASH_TOKEN | division (a comment when followed by / or *) |
* | STAR_TOKEN | multiplication |
~ | TILDE_TOKEN | nothing |
Two characters in a row are always read as two tokens unless they are one of the three pairs above, so == is two = tokens, and && is two & tokens. The line #declare A = 5 % 2; stops with a parse error, because % is not an operator of the language.
The # character
A # is the only character that is taken before the other tokens are handled. When the parser reads a # in a place where a statement may start, it reads the next token and treats it as a directive word: #declare, #if and so on. White space and comments are allowed between the # and the word, so # declare is the directive #declare.
When the token after the # is not a directive word, the # is dropped and that token is put back, to be read as if the # had not been there. So #sphere { 0, 1 } is a sphere, and sphere { 0, 1 #pigment { rgb 1 } } is a sphere with a pigment; the program accepts both without a message.
// a hash before a word that is not a directive
# declare Spaced = 3;
#/* a comment may sit here */ declare Commented = 4;
#debug concat("sum ", str(Spaced + Commented, 0, 0), "\n")sum 7. Both # signs start a #declare although they are not next to the word.Skipped parts
When a conditional branch, a loop that does not run, or a macro body that is only being defined is skipped, the scanner does not cut the text into tokens. It reads on, looking only for three things: a string (so that a # inside a string is not mistaken for a directive), a comment, and a #. Everything else, including characters that are not valid tokens, is ignored. A #end inside a string or comment therefore does not end the skipped part. An unmatched double quote in a skipped part is still an error: Unterminated string literal in input file. The same applies to a block comment that does not end.
#if (0)
@@@ ` \ 0x1G 1.2.3 { ( [ unbalanced
"a string with #end inside"
/* a comment with #end inside */
// a line comment with #end inside
#else
#debug "else branch\n"
#end
#debug "after\n"else branch and after. The first branch holds characters that would be errors anywhere else, and is skipped without a message.Lexical errors
The scanner and the raw tokenizer stop the parse with one of these messages. The message gives the file, the line and the column.
| Message | Cause |
|---|---|
Illegal character c5 in input file.Illegal character U+20ac in input file. | a non-ASCII or control character outside a string or comment; the code is in hexadecimal |
Invalid byte sequence in UTF-8 input file (continuation octet expected). | the file is in UTF-8 (by byte order mark or by detection) and a byte sequence is not valid |
Unterminated block comment in input file. | a /* without its matching */ |
Unterminated string literal in input file. | a " without its closing " |
Illegal escape sequence '\q' in string literal. | an invalid escape sequence in a string that is used as a value |
Invalid numeric literal '2e'. | a number whose text the C library cannot convert |
Parse Error: Expected ... found instead | not a lexical error but the usual result of a character that is a valid token and is in the wrong place, such as @, % or 0x10 |
Not described
Known, not yet covered
- Reference: a
#inside an expression, where a directive is not allowed. The source (Parser::Get_Token) re-issues it as an ordinary token, but a test (#declare A = 1 + #declare B = 2;) ended withExpected 'numeric expression', End of File found instead, and the path to that message was not followed. - Reference: how the other platforms of POV-Ray convert a number such as
2e. The text above is what the program run here does.
Comments
//text/*inside it has no effect/*text*//*inside one opens a deeper level and every*/closes one, so the comment ends at the*/that matches the first/*A block comment that is not closed before the end of the file is an error:
Unterminated block comment in input file.A lone/is the division operator. Comments may stand anywhere between tokens, also between a#and its directive word. A//or/*inside a string literal is part of the string. The scanner has a switch that turns nesting off, but nothing in the parser sets it, so nesting is always on.seen 1. The first line is one comment, because the/*inside it opens a second level that the first*/closes.