There is no separate specification; tokenizer.json is whatever the
Hugging Face tokenizers
library serialises. This page follows its Rust source at release
v0.21.0
(tokenizer/serialization.rs, tokenizer/added_vocabulary.rs,
the model serialization.rs files, utils/truncation.rs,
utils/padding.rs and the component modules), read 2026-11-12. Newer
releases reorganise the source, so later fields may differ.
Top-level keys
One JSON object, written in this order:
| Key | Value |
|---|---|
version | The serialisation version, "1.0". |
truncation | null, or direction, max_length, strategy and stride. |
padding | null, or strategy, direction, pad_to_multiple_of, pad_id, pad_type_id and pad_token. |
added_tokens | The added tokens, in ascending ID order. |
normalizer | null or a normaliser object. |
pre_tokenizer | null or a pre-tokeniser object. |
post_processor | null or a post-processor object. |
decoder | null or a decoder object. |
model | The model, with its vocabulary. |
The component objects carry a type naming the component. The library
has normalisers such as NFC, NFD, NFKC, NFKD, lower-casing, accent stripping, BERT,
replace, prepend, byte-level, precompiled and sequences of others; pre-tokenisers
such as BERT, byte-level, whitespace, punctuation, digits, Metaspace, split,
Unicode scripts and sequences; post-processors for BERT, RoBERTa, byte-level,
templates and sequences; and decoders such as BPE, byte-level, byte fallback, CTC,
fuse, Metaspace, replace, strip, WordPiece and sequences.
Added tokens
| Field | Meaning |
|---|---|
id | The token’s ID. |
content | The token text. |
single_word | Whether it must be a whole word or can match inside words. |
lstrip, rstrip | Whether to strip white space on its left or right. |
normalized | Whether it is matched after normalisation. |
special | Whether it is a special token. |
Models
type | Fields, in order |
|---|---|
BPE | dropout, unk_token, continuing_subword_prefix, end_of_word_suffix, fuse_unk, byte_fallback, ignore_merges, vocab (token to ID, in ID order), merges (pairs of tokens in rank order, written as two-element arrays; the older form "a b" as one string is still read). |
WordPiece | unk_token, continuing_subword_prefix, max_input_chars_per_word, vocab. |
Unigram | unk_id, vocab (a list of [piece, score] pairs), byte_fallback. |
WordLevel | vocab, unk_token. |