.EL File Format – Native compilation

Requirements and history

Emacs has to be built with native compilation, which needs libgccjit. To compile Lisp, the machine also needs GCC and Binutils (the assembler and the linker). A Lisp program can ask whether the running Emacs can produce and load native code with native-comp-available-p; on systems that load libgccjit dynamically it also checks that the library can be loaded.

The pipeline

Native compilation is implemented as a side effect of byte-compilation, so compiling a file natively always produces its byte-code too, and everything in Byte-compilation about warnings, macros and eval-when-compile applies. comp.el then runs a list of passes in order:

1. the front end spills the byte compiler’s intermediate form (LAP) for each function;
2. the code is turned into LIMPLE, the compiler’s own intermediate representation;
3. optimisation passes run on it: forward propagation, call optimisation, an interprocedural purity analysis, type constraints and type checks, tail-call optimisation, and the computation of function types;
4. the final pass builds the code with libgccjit, which calls GCC to produce and link the shared library.

The last step runs in a separate subprocess that starts the same Emacs executable. Messages are written to a buffer *Native-compile-Log*, which uses native-comp-limple-mode (like Fundamental mode but with highlighting for the log). In batch mode they are printed with message. native-comp-verbose sets how much is logged: 0 nothing, 1 the final LIMPLE, 2 also the LAP and extra pass information, 3 everything.

Commands and functions

emacs --batch -f batch-native-compile foo.el

The eln-cache

native-comp-eln-load-path is the list of directories where Emacs looks for .eln files, in the same role as load-path for source. A relative name is taken relative to invocation-directory. The last entry is the system directory with the files installed by the Emacs build. The default user entry is the eln-cache directory inside user-emacs-directory. Emacs writes new files into the first writable directory of the list.

Inside each directory Emacs looks only in one subdirectory. Its name, stored in comp-native-version-dir, is built from the Emacs version and an 8-character hash that depends on the native-compilation ABI, so builds that cannot share code never share a directory. The files in it are named after the source file:

~/.emacs.d/eln-cache/30.1-1a2b3c4d/foo-0123abcd-4567ef89.eln

This path is an illustration of the shape. Per the cleaning code in comp.el, a file name is the library name, a dash, a hexadecimal hash, a dash, a second hexadecimal hash and .eln. The first hash identifies the source file: when a library is recompiled, Emacs removes every .eln in the load path that has the same first hash, which deletes the stale versions. comp-lookup-eln finds the compiled file for a source file by turning its name into a relative .eln name and testing each directory of the load path. The manual notes that because of the unique names, files made by several Emacs versions can live in one directory, and a different version will normally produce a differently named file that only it can load.

When Emacs loads a .elc found on load-path, it looks for a matching .eln in this cache. The source file must still be present, because the lookup and the background compilation both use it; native-comp-warning-on-missing-source (default on) warns when it is not. If load-prefer-newer is on, load does not take the .eln.

To keep the cache out of the home directory, change native-comp-eln-load-path, call startup-redirect-eln-cache, or point HOME at a directory that does not exist. Trampolines may still produce a few files.

Asynchronous (JIT) compilation

When a .elc file is loaded and no .eln exists, Emacs can compile it in the background. This happens when native-comp-jit-compilation is non-nil. The work is done by separate batch Emacs subprocesses; when one finishes, the new .eln is loaded and its functions replace the ones from the .elc. Because the subprocess starts from a fresh Emacs, it can report warnings that a normal byte-compile does not, for example for a missing require.

Trampolines

A trampoline is a small piece of native code that lets natively compiled code call a primitive (a function written in C) that has been advised or redefined. Calls from compiled code to primitives are normally direct, so they would skip the advice. comp-trampoline-compile in comp.el builds one from a synthetic lambda that calls the primitive through funcall or apply, compiled with native-comp-speed 1 and without byte optimisation, so the call is not optimised away. It is stored as its own .eln file.

native-comp-enable-subr-trampolines controls this. t (default) generates them, even when JIT compilation is off. nil turns generation off, and then calls from native code to an advised primitive ignore the advice unless a trampoline already exists. A string names a directory, relative to invocation-directory, where trampolines are stored, outside the version subdirectory, and not found again in later sessions. When no writable directory is available, a trampoline goes to temporary-file-directory.

Optimisation settings

The same settings apply per function through a declare form:

(defun my-hot-loop (n)
  (declare (speed 3) (safety 1))
  (let ((sum 0))
    (dotimes (i n) (setq sum (+ sum i)))
    sum))

(speed -1) makes native compilation emit byte-code for that function. In Emacs 30 a declaration (ftype TYPE) states the function’s type, which the native compiler can use for better code when compilation-safety is 0.

The file variable no-native-compile stops only native compilation of a file. no-byte-compile stops both and wins if both are set. native-comp-bootstrap-deny-list lists regexps of files to skip while building Emacs.

What an .eln file is

A .eln file is a native shared library, the kind of file the system uses for .so or .dll: ELF on Linux, Mach-O on macOS and PE on Windows. It holds the machine code of one Lisp file’s functions, in the form the building Emacs expects, with entry points that depend on Emacs’s internal primitives and calling conventions. The manual states the consequences: it is as portable as any shared library, so it works on the same machine or very similar ones (same processor and run-time libraries), and an Emacs of an earlier or later version will usually not load it. Native compilation of the same code by a different Emacs gives a different file name.

At speed -1 the file is still written, but contains the code as byte-code. The byte-compiled .elc remains needed as a fallback and as the source of the lookup.

In Lisp, a native-compiled function prints as #<subr foo>, like a primitive (this is recalled, not checked against the sources). native-comp-function-p (formerly subr-native-elisp-p) tests for one, and primitive-function-p or subr-primitive-p tests for a built-in written in C. symbol-file with its third argument reports the .eln that defined a symbol (Emacs 29).

How the viewer treats .eln

The viewer names the file “Natively compiled Emacs Lisp (eln)” and shows it as bytes only. Because the file is a shared library, a general file recogniser may identify it as ELF, Mach-O or PE (a DLL) instead. That is correct for the container; the name above describes the role of the file.

Not described

Known, not yet covered

• Reference: how the two hashes in a file name are computed, and the name of a trampoline file (comp-trampoline-filename). They are defined in comp.c, which was not among the downloaded sources.
• Reference: the deny lists for JIT compilation (native-comp-jit-compilation-deny-list and, in older releases, native-comp-deferred-compilation-deny-list) and the startup hooks that start the JIT. They are in comp-run.el, not downloaded.
• Reference: the internal layout of an .eln (symbols, relocation tables, the embedded Lisp data) and the individual compiler passes.
• Reference: the printed form of a native-compiled function was recalled, not confirmed in the sources.

See also

• Byte-compilation – the front end and the .elc format
• Emacs versions – what each release changed
• Functions – the declare form
• Compilation of Lisp to Native Code – the manual’s chapter