.TEX File Format – Characters and commands

LaTeX and TeX source

A .tex file is not a data format with fixed fields but a program in TeX’s macro language: what a line means depends on the definitions in force when TeX reads it, and those can change part-way through. What is fixed is how TeX turns characters into tokens, described by Donald Knuth in The TeXbook and in the source of TeX itself. This page covers that layer; the LaTeX page covers the structure almost all .tex files follow today.

Encoding and lines

  • Files are plain text. Since the release of April 2018, LaTeX takes UTF-8 as the default input encoding with pdfTeX and classic TeX; LuaTeX and XeTeX have always read UTF-8 only (LaTeX News 28). Files in another encoding, such as Latin-1, declare it with an option to the inputenc package (\usepackage[latin1]{inputenc}); older files that relied on raw 8-bit input need \UseRawInputEncoding as their first line.
  • TeX reads line by line. It removes trailing spaces from each line (§31 of tex.web) and appends the \endlinechar, by default the carriage return (code 13, category 5), so a line end acts like a space. How line ends are recognised in the file is left to the implementation.

Category codes

Every character has a category code that decides how TeX treats it. The codes can be changed with \catcode, which is how verbatim text and many packages work. TeX itself (IniTeX, before any format is loaded) sets only a few of them; plain TeX sets the rest, and LaTeX keeps the same assignments:

CodeMeaningSet by TeXSet by plain TeX
0Escape character: starts a command\
1Begin group{
2End group}
3Maths shift$
4Alignment tab&
5End of linecarriage return (code 13)
6Macro parameter#
7Superscript^ and ^^K (code 11)
8Subscript_ and ^^A (code 1)
9Ignorednull (code 0)
10Spacespace (code 32)tab (code 9)
11LetterA–Z, a–z(LaTeX also makes @ a letter in package and class files)
12Otherevery character not listed here
13Active character: a one-character command~ (a tie, a non-breaking space) and ^^L (form feed, defined as \outer\par)
14Comment: the rest of the line, and its line end, is ignored%
15Invalid: an errordelete (code 127)

So the ten characters # $ % & ~ _ ^ \ { } are special in text. In LaTeX seven of them are typed with a backslash in front (\#, \$, \%, \&, \_, \{, \}); the LaTeX2e reference gives \~{} and \^{} for a tilde and circumflex and \textbackslash{} for a backslash.

Commands

  • Control word: the escape character followed by one or more letters (category 11), such as \section. Spaces after it are skipped, so \TeX{} or \TeX\ is needed to keep one.
  • Control symbol: the escape character followed by any single non-letter, such as \$ or \\. Spaces after it are kept, except after \ (a control space).
  • Active character: a character of category 13, a command by itself.
  • Arguments are single tokens or groups in braces. LaTeX adds optional arguments in square brackets and starred forms (\section*).

Spaces and paragraphs

  • Several spaces or tabs in a row count as one.
  • Spaces at the start of a line are ignored.
  • A single line end counts as a space.
  • An empty line (two line ends in a row) ends a paragraph: TeX reads it as the command \par. Further empty lines add nothing.
  • A % at the end of a line removes the line end, which is used to avoid unwanted spaces in macro code.

Groups

Braces { } make a group: settings made inside, such as a font change, end at the closing brace. \begingroup and \endgroup do the same, and LaTeX environments (\begin{…} … \end{…}) are groups too. Braces must balance in a file and inside every argument.

The ^^ notation

Two identical superscript characters (category 7) followed by a character write a character that is hard to type (§352 of tex.web): ^^ plus two lower-case hexadecimal digits (0–9, a–f) is that code (^^e9 is 0xE9); ^^ plus any other character with a code below 128 adds 64 to it if the code is below 64 and subtracts 64 otherwise (^^M is carriage return, ^^? is delete, ^^@ is null). The conversion also happens inside command names.

Macro definitions

CommandMeaning
\def\name#1#2{…#1…#2…}TeX primitive: defines a macro with up to nine parameters, #1 to #9. Text between parameters in the definition must match in use (delimited parameters).
\edef, \gdef, \xdefExpand the body when defining; define globally; both.
\let\a=\bMakes \a a copy of \b’s current meaning.
\newcommand{\name}[2][default]{…}LaTeX: defines a new command with 0 to 9 arguments, counting the optional one; with [default] the first is optional. An error if the name is already defined. \renewcommand redefines an existing command (an error if it is undefined); \providecommand defines it only if it does not exist yet.
\NewDocumentCommand{\name}{O{x} m}{…}LaTeX kernel: the arguments are given as a specification, such as m mandatory, o optional, O{default} optional with a default, s star, and others. An error if the name is already defined.
\newenvironment{name}[n]{begin code}{end code}LaTeX: defines an environment.

Inside a definition, ## stands for a # in a nested definition. Names containing @ are internal: they can be used only between \makeatletter and \makeatother in a document, but freely in .sty and .cls files.

Magic comments

Comments at the top of a file that editors and build tools read, not TeX itself. The form began in TeXShop; the TeXstudio manual lists these, among others:

% !TeX program = xelatex
% !TeX encoding = utf8
% !TeX root = ../thesis.tex
% !TeX spellcheck = en_GB
% !BIB program = biber

TeXstudio reads % !BIB program for compatibility with TeXShop and TeXworks.

Plain TeX files

A file for plain TeX, Knuth’s own macro set, has no preamble: it uses commands such as \centerline, \bf and \item directly, and ends with \bye, which plain.tex defines as \par\vfill\supereject\end. Plain TeX and LaTeX files both use the .tex extension, so the first commands of a file are the way to tell them apart.

Sources