.ENGINE

Introduction

A TensorRT engine is a model compiled by NVIDIA TensorRT to run as fast as possible on one kind of NVIDIA GPU. TensorRT takes a trained network, usually as an ONNX file, merges layers, picks the fastest GPU kernels for each step and can lower the number precision, then writes the result out as a serialised engine. The extension is .engine or .plan; NVIDIA’s Triton Inference Server expects the name model.plan.

An engine is not a portable model. It works only with the TensorRT version that built it and on the same GPU model, or at least the same GPU architecture, so it is normally built on the machine that will run it and kept beside the ONNX file it came from.

Structure

The layout is internal to TensorRT and not published. The file holds the optimised network, the chosen kernels and the weights, and is read back with TensorRT’s runtime (deserializeCudaEngine). The trtexec tool that comes with TensorRT builds one with --saveEngine and can load and time one with --loadEngine.

Links