.JSON

Hugging Face tokenizer

Introduction

A language model does not read text directly. A tokenizer first splits the text into tokens, whole words or pieces of words, and turns each into a number from the model’s vocabulary. tokenizer.json is the file that holds a complete tokenizer for Hugging Face’s tokenizers library: every step of the process, the vocabulary and the rules. Nearly every language model on Hugging Face ships one beside its weights.

It usually comes with tokenizer_config.json, which names the tokenizer class, the special tokens and often the chat template, and sometimes special_tokens_map.json. Older models used vocab.txt, vocab.json and merges.txt, or a SentencePiece tokenizer.model, instead.

Structure

The file is one JSON object. Its keys follow the steps of tokenising in order:

  • version: the version of the file format
  • truncation and padding: default settings, or null
  • added_tokens: special tokens such as the start and end of text, each with its ID and content
  • normalizer: clean-up of the text first, such as Unicode normalisation or lower case
  • pre_tokenizer: the first split, for example on spaces and punctuation, or into bytes
  • model: the algorithm and its data; type is BPE, WordPiece, Unigram or WordLevel, vocab maps each token to its ID, and a BPE model also lists its merges
  • post_processor: tokens added around the result, such as a start token
  • decoder: how to turn tokens back into text

The vocabulary and merges make the file large, often several megabytes for a model with a vocabulary of 100,000 tokens or more.

Also used for

Links

Software

Developers

  • tokenizers, written in Rust, with bindings for Python and Node.js
  • Transformers.js, JavaScript, which reads the same files in the browser

Reference