Tokenization in Transformers V5: Simpler, Clearer, and More Modular

•December 18, 2025

Hugging Face•Dec 18, 2025

Companies Mentioned

Google

GOOG

OpenAI

Why It Matters

The redesign reduces code duplication, speeds up preprocessing, and empowers developers to train and tweak tokenizers without opaque black‑box constraints, accelerating LLM deployment and innovation.

Key Takeaways

•One file per model replaces fast/slow duplication.
•Architecture exposed via properties for inspection.
•Training tokenizers now single-call with train_new_from_iterator.
•AutoTokenizer auto-selects correct class, simplifying usage.
•Rust backend provides speed, Python fallback for flexibility.

Pulse Analysis

Tokenization remains the gateway between raw text and large language models, yet its complexity has long hidden crucial decisions behind opaque files. In earlier versions of the Transformers library, each model shipped with separate "fast" and "slow" tokenizer implementations, and the underlying architecture—normalizers, pre‑tokenizers, and decoders—was buried in binary checkpoints. This made debugging, customization, and especially training new tokenizers a cumbersome, error‑prone process, limiting developers who needed fine‑grained control over text preprocessing.

Version 5 addresses those pain points by adopting an "architecture‑first" philosophy. A single Python file now defines the entire tokenizer stack, exposing the normalizer, pre‑tokenizer, model, post‑processor, and decoder as direct attributes. The default Rust‑based TokenizersBackend delivers high‑throughput performance, while a PythonBackend remains available for niche custom logic. Crucially, the new API lets users instantiate a blank tokenizer and train it from scratch with a single call—`train_new_from_iterator`—eliminating the manual pipeline reconstruction that plagued v4. The hierarchy is clearer: `PreTrainedTokenizerBase` provides a unified interface, `TokenizersBackend` wraps the fast Rust engine, and specialized backends (SentencePiece, Python) handle edge cases.

For enterprises and research teams, these changes translate into faster development cycles and lower maintenance overhead. Engineers can now inspect and modify tokenization components without digging into compiled artifacts, enabling rapid experimentation with domain‑specific vocabularies or novel preprocessing strategies. The streamlined `AutoTokenizer` continues to abstract model‑specific details, reducing onboarding friction for new users. Overall, v5’s modular, transparent design positions the Transformers ecosystem to scale with the growing demand for customized LLM deployments, while preserving the speed advantages of the Rust backend.

Tokenization in Transformers v5: Simpler, Clearer, and More Modular

Transformers v5 redesigns how tokenizers work

TL;DR: This blog explains how tokenization works in Transformers and why v5 is a major redesign, with clearer internals, a clean class hierarchy, and a single fast backend. It’s a practical guide for anyone who wants to understand, customize, or train model‑specific tokenizers instead of treating them as black boxes.

What is Tokenization?
The Tokenization Pipeline
Tokenization Algorithms
Accessing tokenizers through transformers
The Tokenizer Class Hierarchy in transformers
AutoTokenizer Automatically Selects the Correct Tokenizer Class
v5 Separates Tokenizer Architecture from Trained Vocab
Summary

For experts: If you're already familiar with the concepts and want to understand the changes in v5, go to v5 Separates Tokenizer Architecture from Trained Vocab.

What is tokenization?

Language models don't read raw text. They consume sequences of integers usually called token IDs or input IDs. Tokenization is the process of converting raw text into these token IDs. (Try the tokenization playground here to visualize tokenization.)

Tokenization is a broad concept used across natural language processing and text processing generally. This post focuses specifically on tokenization for Large Language Models (LLMs) using the transformers and tokenizers libraries.


from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B")



text = "Hello world"

tokens = tokenizer(text)



print(tokens["input_ids"])

# [9906, 1917]



print(tokenizer.convert_ids_to_tokens(tokens["input_ids"]))

# ['Hello', 'Ġworld']

Ġworld (above) is a single token that represents the character sequence " world" (with the leading space).

A token is the smallest string unit the model sees. It can be a character, word, or sub‑word chunk like "play" or "##ing" (the ## pattern is a legacy from WordPiece). The vocabulary maps each unique token to a token ID.


from transformers import AutoTokenizer



tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B")

print(tokenizer.vocab)



# {'ÎĹÎľ': 106502, 'ĠPeel': 89694, '.languages': 91078, ...}

A good tokenizer compresses text into the smallest possible number of tokens. Fewer tokens mean more usable context without increasing model size. Training a tokenizer boils down to finding the best compression rules for your dataset. For example, if you work on a Chinese corpus you can sometimes find very nice surprises 😉.

The tokenization pipeline

Tokenization happens in stages. Each stage transforms the text before passing it to the next:

| Stage | Purpose | Example |

|------------------|----------------------------------------------------|-------------------------------------------|

| Normalizer | Standardizes text (lowercasing, Unicode normalization, whitespace cleanup) | "HELLO World" → "hello world" |

| Pre‑tokenizer| Splits text into preliminary chunks | "hello world" → ["hello", " world"] |

| Model | Applies the tokenization algorithm (BPE, Unigram, etc.) | ["hello", " world"] → [9906, 1917] |

| Post‑processor| Adds special tokens (BOS, EOS, padding) | [9906, 1917] → [1, 9906, 1917, 2] |

| Decoder | Converts token IDs back to text | [9906, 1917] → "hello world" |

Each component is independent; you can swap normalizers or change the algorithm without rewriting the whole pipeline.

You can access the Rust‑based tokenizer through _tokenizer. We go in more depth about it in the TokenizersBackend section.


from transformers import AutoTokenizer



tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-270m-it")



print(f"{tokenizer._tokenizer.normalizer=}")

# Replace(...)



print(f"{tokenizer._tokenizer.pre_tokenizer=}")

# Split(...)



print(f"{tokenizer._tokenizer.model=}")

# BPE(...)



print(f"{tokenizer._tokenizer.post_processor=}")

# TemplateProcessing(...)



print(f"{tokenizer._tokenizer.decoder=}")

# Sequence(decoders=[Replace(...), ByteFallback(), Fuse()])

Tokenization algorithms

The following algorithms dominate modern language‑model tokenizers:

Byte Pair Encoding (BPE) – iteratively merges the most frequent character pairs. Deterministic and widely used.


from transformers import AutoTokenizer



tokenizer = AutoTokenizer.from_pretrained("openai/gpt-oss-20b")

print(tokenizer._tokenizer.model)   # BPE(...)

Unigram – probabilistic approach that selects the most likely segmentation from a large initial vocabulary. More flexible than strict BPE.


from transformers import AutoTokenizer



tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-base")

print(tokenizer._tokenizer.model)   # Unigram(...)

WordPiece – resembles BPE but uses different merge criteria based on likelihood.


from transformers import AutoTokenizer



tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

print(tokenizer._tokenizer.model)   # WordPiece(...)

Accessing tokenizers through `transformers`

The tokenizers library (Rust‑based) is fast, efficient, and language‑model agnostic. It handles the mechanics of converting text into token IDs and back, but it does not implement model‑specific conventions (e.g., chat templates, special tokens).


from tokenizers import Tokenizer



tokenizer = Tokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B")

text = "Hello world"

encodings = tokenizer.encode(text)



print(encodings.ids)      # [9906, 1917]

print(encodings.tokens)   # ['Hello', 'Ġworld']

The raw tokenizers output lacks the formatting required by conversational models. The transformers library bridges this gap by wrapping the raw backend and adding model‑aware functionality.


from transformers import AutoTokenizer



tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B")



# Format a conversation using the model's chat template

prompt = "Give me a brief explanation of gravity in simple terms."

messages = [{"role": "user", "content": prompt}]

text = tokenizer.apply_chat_template(

    messages,

    tokenize=False,

    add_generation_prompt=True,

)



print(text)

# <|im_start|>system

# ...

# <|im_start|>user

# Give me a brief explanation of gravity in simple terms.<|im_end|>

# <|im_start|>assistant



model_inputs = tokenizer([text], return_tensors="pt")

The transformers tokenizer adds:

Chat template application – inserts model‑specific special tokens.
Automatic special‑token insertion – BOS/EOS handling.
Truncation to context length – respects model’s maximum sequence length.
Batch encoding with padding – consistent padding token and direction.
Return‑format options – PyTorch tensors, NumPy arrays, etc.

transformers implements the tokenization API most commonly used in the ML community (encode, decode, convert_tokens_to_ids, …).

The tokenizer class hierarchy in `transformers`

The library organizes tokenizers into a clear class hierarchy. At the top sits a base class that defines the common interface; backend classes handle the actual tokenization; model‑specific classes configure the backends.

Class hierarchy

`PreTrainedTokenizerBase` – common interface for all tokenizers

Special‑token properties – bos_token, eos_token, pad_token, unk_token.
Encoding interface – __call__, encode, encode_plus.
Decoding interface – decode, batch_decode.
Serialization – save_pretrained, from_pretrained.
Chat‑template support – apply_chat_template.

All tokenizers ultimately inherit from this base class, ensuring consistent behavior.

`TokenizersBackend` – wraps the Rust `tokenizers` library

Stores the Rust tokenizer object (self._tokenizer).
Delegates heavy work to the Rust backend while the Python wrapper adds model‑aware features.

Typical subclasses: LlamaTokenizer, GemmaTokenizer.

`PythonBackend` – pure‑Python mixin

Used when custom tokenization logic or legacy compatibility is required.
Slower than the Rust backend but more flexible.

Examples: CTRLTokenizer, CanineTokenizer.

`SentencePieceBackend` – handles SentencePiece models

Wraps Google’s SentencePiece library.
Used by models such as SiglipTokenizer and BartphoTokenizer.

`AutoTokenizer` automatically selects the correct tokenizer class


from transformers import AutoTokenizer



tokenizer = AutoTokenizer.from_pretrained("gpt2")

AutoTokenizer:

Downloads tokenizer_config.json.
Reads the model type (e.g., "gpt2").
Looks up the appropriate tokenizer class via TOKENIZER_MAPPING_NAMES.
Instantiates and returns the configured tokenizer.

The benefit is that you never need to know the exact tokenizer class for a model; AutoTokenizer does it for you.

v5 separates tokenizer architecture from trained vocab

The problem with v4

Tokenizers were opaque black boxes tied to checkpoint files.
No easy way to inspect architecture (normalizer, model type, etc.).
Two parallel implementations per model (*_Tokenizer and *_TokenizerFast) caused code duplication, behavioral discrepancies, and user confusion.
Training a tokenizer from scratch required manual reconstruction of the pipeline.

The v5 solution

Architecture first, parameters later – just like nn.Module.
One file per model, defaulting to the Rust‑backed TokenizersBackend.
Clear, inspectable properties (tokenizer.normalizer, tokenizer.model, …).
Simple API to train from scratch (tokenizer.train_new_from_iterator, tokenizer.train).

Example: training a LLaMA‑style tokenizer from scratch


from transformers import LlamaTokenizer

from datasets import load_dataset



# Initialize a blank tokenizer (architecture only)

tokenizer = LlamaTokenizer()



dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")



def get_training_corpus():

    batch = 1000

    for i in range(0, len(dataset), batch):

        yield dataset[i : i + batch]["text"]



trained_tokenizer = tokenizer.train_new_from_iterator(

    text_iterator=get_training_corpus(),

    vocab_size=32_000,

    length=len(dataset),

    show_progress=True,

)



trained_tokenizer.push_to_hub("my_custom_tokenizer")

tokenizer = LlamaTokenizer.from_pretrained("my_custom_tokenizer")

The resulting tokenizer behaves exactly like the official LLaMA tokenizer but with a custom vocabulary.

| Aspect | v4 | v5 |

|--------|----|----|

| Files per model | Two (tokenization_X.py, tokenization_X_fast.py) | One (tokenization_X.py) |

| Default backend | Split between Python and Rust | Rust (TokenizersBackend) preferred |

| Architecture visibility | Hidden in serialized files | Explicit in class definition |

| Training from scratch | Manual pipeline construction | tokenizer.train(...) / train_new_from_iterator |

| Component inspection | Difficult, undocumented | Direct properties (tokenizer.normalizer, etc.) |

| Parent classes | PreTrainedTokenizer, PreTrainedTokenizerFast | TokenizersBackend (or SentencePieceBackend, PythonBackend) |

Summary

Transformers v5 brings three key improvements to tokenization:

One file per model – no more slow/fast duplication.
Visible architecture – you can inspect normalizers, pre‑tokenizers, decoders, and model type directly.
Trainable templates – instantiate a tokenizer architecture and train it on your own data with a single call.

The wrapper layer between tokenizers and transformers remains essential: it adds model awareness (chat templates, special tokens, context‑length handling) that raw tokenization alone does not provide. v5 simply makes that layer clearer and more customizable.

Further resources

Let's build the GPT Tokenizer – https://youtu.be/zduSFxRajkE?si=ZAfCjZjpyPHsnyfF
Gotchas in Tokenizer Behavior Every Developer Should Know – https://huggingface.co/blog/qgallouedec/gotchas-in-tokenizer-behavior
Chat Templates – https://huggingface.co/blog/chat-templates
Community‑curated resource list – https://x.com/ariG23498/status/1999058214906888237

Read Original Article

Comments

Want to join the conversation?

Loading comments...

Transformers v5 redesigns how tokenizers work

What is Tokenization?
The Tokenization Pipeline
Tokenization Algorithms
Accessing tokenizers through transformers
The Tokenizer Class Hierarchy in transformers
AutoTokenizer Automatically Selects the Correct Tokenizer Class
v5 Separates Tokenizer Architecture from Trained Vocab
Summary

For experts: If you're already familiar with the concepts and want to understand the changes in v5, go to v5 Separates Tokenizer Architecture from Trained Vocab.

What is tokenization?


from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B")



text = "Hello world"

tokens = tokenizer(text)



print(tokens["input_ids"])

# [9906, 1917]



print(tokenizer.convert_ids_to_tokens(tokens["input_ids"]))

# ['Hello', 'Ġworld']

Ġworld (above) is a single token that represents the character sequence " world" (with the leading space).


from transformers import AutoTokenizer



tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B")

print(tokenizer.vocab)



# {'ÎĹÎľ': 106502, 'ĠPeel': 89694, '.languages': 91078, ...}

The tokenization pipeline

Tokenization happens in stages. Each stage transforms the text before passing it to the next:

| Stage | Purpose | Example |

|------------------|----------------------------------------------------|-------------------------------------------|

| Normalizer | Standardizes text (lowercasing, Unicode normalization, whitespace cleanup) | "HELLO World" → "hello world" |

| Pre‑tokenizer| Splits text into preliminary chunks | "hello world" → ["hello", " world"] |

| Model | Applies the tokenization algorithm (BPE, Unigram, etc.) | ["hello", " world"] → [9906, 1917] |

| Post‑processor| Adds special tokens (BOS, EOS, padding) | [9906, 1917] → [1, 9906, 1917, 2] |

| Decoder | Converts token IDs back to text | [9906, 1917] → "hello world" |

Each component is independent; you can swap normalizers or change the algorithm without rewriting the whole pipeline.

You can access the Rust‑based tokenizer through _tokenizer. We go in more depth about it in the TokenizersBackend section.


from transformers import AutoTokenizer



tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-270m-it")



print(f"{tokenizer._tokenizer.normalizer=}")

# Replace(...)



print(f"{tokenizer._tokenizer.pre_tokenizer=}")

# Split(...)



print(f"{tokenizer._tokenizer.model=}")

# BPE(...)



print(f"{tokenizer._tokenizer.post_processor=}")

# TemplateProcessing(...)



print(f"{tokenizer._tokenizer.decoder=}")

# Sequence(decoders=[Replace(...), ByteFallback(), Fuse()])

Tokenization algorithms

The following algorithms dominate modern language‑model tokenizers:

Byte Pair Encoding (BPE) – iteratively merges the most frequent character pairs. Deterministic and widely used.


from transformers import AutoTokenizer



tokenizer = AutoTokenizer.from_pretrained("openai/gpt-oss-20b")

print(tokenizer._tokenizer.model)   # BPE(...)

Unigram – probabilistic approach that selects the most likely segmentation from a large initial vocabulary. More flexible than strict BPE.


from transformers import AutoTokenizer



tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-base")

print(tokenizer._tokenizer.model)   # Unigram(...)

WordPiece – resembles BPE but uses different merge criteria based on likelihood.


from transformers import AutoTokenizer



tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

print(tokenizer._tokenizer.model)   # WordPiece(...)

Accessing tokenizers through `transformers`


from tokenizers import Tokenizer



tokenizer = Tokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B")

text = "Hello world"

encodings = tokenizer.encode(text)



print(encodings.ids)      # [9906, 1917]

print(encodings.tokens)   # ['Hello', 'Ġworld']

The raw tokenizers output lacks the formatting required by conversational models. The transformers library bridges this gap by wrapping the raw backend and adding model‑aware functionality.


from transformers import AutoTokenizer



tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B")



# Format a conversation using the model's chat template

prompt = "Give me a brief explanation of gravity in simple terms."

messages = [{"role": "user", "content": prompt}]

text = tokenizer.apply_chat_template(

    messages,

    tokenize=False,

    add_generation_prompt=True,

)



print(text)

# <|im_start|>system

# ...

# <|im_start|>user

# Give me a brief explanation of gravity in simple terms.<|im_end|>

# <|im_start|>assistant



model_inputs = tokenizer([text], return_tensors="pt")

The transformers tokenizer adds:

Chat template application – inserts model‑specific special tokens.
Automatic special‑token insertion – BOS/EOS handling.
Truncation to context length – respects model’s maximum sequence length.
Batch encoding with padding – consistent padding token and direction.
Return‑format options – PyTorch tensors, NumPy arrays, etc.

transformers implements the tokenization API most commonly used in the ML community (encode, decode, convert_tokens_to_ids, …).

The tokenizer class hierarchy in `transformers`

Class hierarchy

`PreTrainedTokenizerBase` – common interface for all tokenizers

Special‑token properties – bos_token, eos_token, pad_token, unk_token.
Encoding interface – __call__, encode, encode_plus.
Decoding interface – decode, batch_decode.
Serialization – save_pretrained, from_pretrained.
Chat‑template support – apply_chat_template.

All tokenizers ultimately inherit from this base class, ensuring consistent behavior.

`TokenizersBackend` – wraps the Rust `tokenizers` library

Stores the Rust tokenizer object (self._tokenizer).
Delegates heavy work to the Rust backend while the Python wrapper adds model‑aware features.

Typical subclasses: LlamaTokenizer, GemmaTokenizer.

`PythonBackend` – pure‑Python mixin

Used when custom tokenization logic or legacy compatibility is required.
Slower than the Rust backend but more flexible.

Examples: CTRLTokenizer, CanineTokenizer.

`SentencePieceBackend` – handles SentencePiece models

Wraps Google’s SentencePiece library.
Used by models such as SiglipTokenizer and BartphoTokenizer.

`AutoTokenizer` automatically selects the correct tokenizer class


from transformers import AutoTokenizer



tokenizer = AutoTokenizer.from_pretrained("gpt2")

AutoTokenizer:

Downloads tokenizer_config.json.
Reads the model type (e.g., "gpt2").
Looks up the appropriate tokenizer class via TOKENIZER_MAPPING_NAMES.
Instantiates and returns the configured tokenizer.

The benefit is that you never need to know the exact tokenizer class for a model; AutoTokenizer does it for you.

v5 separates tokenizer architecture from trained vocab

The problem with v4

Tokenizers were opaque black boxes tied to checkpoint files.
No easy way to inspect architecture (normalizer, model type, etc.).
Two parallel implementations per model (*_Tokenizer and *_TokenizerFast) caused code duplication, behavioral discrepancies, and user confusion.
Training a tokenizer from scratch required manual reconstruction of the pipeline.

The v5 solution

Architecture first, parameters later – just like nn.Module.
One file per model, defaulting to the Rust‑backed TokenizersBackend.
Clear, inspectable properties (tokenizer.normalizer, tokenizer.model, …).
Simple API to train from scratch (tokenizer.train_new_from_iterator, tokenizer.train).

Example: training a LLaMA‑style tokenizer from scratch


from transformers import LlamaTokenizer

from datasets import load_dataset



# Initialize a blank tokenizer (architecture only)

tokenizer = LlamaTokenizer()



dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")



def get_training_corpus():

    batch = 1000

    for i in range(0, len(dataset), batch):

        yield dataset[i : i + batch]["text"]



trained_tokenizer = tokenizer.train_new_from_iterator(

    text_iterator=get_training_corpus(),

    vocab_size=32_000,

    length=len(dataset),

    show_progress=True,

)



trained_tokenizer.push_to_hub("my_custom_tokenizer")

tokenizer = LlamaTokenizer.from_pretrained("my_custom_tokenizer")

The resulting tokenizer behaves exactly like the official LLaMA tokenizer but with a custom vocabulary.

| Aspect | v4 | v5 |

|--------|----|----|

| Files per model | Two (tokenization_X.py, tokenization_X_fast.py) | One (tokenization_X.py) |

| Default backend | Split between Python and Rust | Rust (TokenizersBackend) preferred |

| Architecture visibility | Hidden in serialized files | Explicit in class definition |

| Training from scratch | Manual pipeline construction | tokenizer.train(...) / train_new_from_iterator |

| Component inspection | Difficult, undocumented | Direct properties (tokenizer.normalizer, etc.) |

| Parent classes | PreTrainedTokenizer, PreTrainedTokenizerFast | TokenizersBackend (or SentencePieceBackend, PythonBackend) |

Summary

Transformers v5 brings three key improvements to tokenization:

One file per model – no more slow/fast duplication.
Visible architecture – you can inspect normalizers, pre‑tokenizers, decoders, and model type directly.
Trainable templates – instantiate a tokenizer architecture and train it on your own data with a single call.

Further resources

Let's build the GPT Tokenizer – https://youtu.be/zduSFxRajkE?si=ZAfCjZjpyPHsnyfF
Gotchas in Tokenizer Behavior Every Developer Should Know – https://huggingface.co/blog/qgallouedec/gotchas-in-tokenizer-behavior
Chat Templates – https://huggingface.co/blog/chat-templates
Community‑curated resource list – https://x.com/ariG23498/status/1999058214906888237

AI Pulse

Tokenization in Transformers V5: Simpler, Clearer, and More Modular

Companies Mentioned

Why It Matters

Key Takeaways

Pulse Analysis

Ask Pulse AI:

Tokenization in Transformers v5: Simpler, Clearer, and More Modular

Transformers v5 redesigns how tokenizers work

Table of Contents

What is tokenization?

The tokenization pipeline

Tokenization algorithms

Accessing tokenizers through transformers

The tokenizer class hierarchy in transformers

PreTrainedTokenizerBase – common interface for all tokenizers

TokenizersBackend – wraps the Rust tokenizers library

PythonBackend – pure‑Python mixin

SentencePieceBackend – handles SentencePiece models

AutoTokenizer automatically selects the correct tokenizer class

v5 separates tokenizer architecture from trained vocab

The problem with v4

The v5 solution

Example: training a LLaMA‑style tokenizer from scratch

Summary

Further resources

Comments

AI Pulse

Tokenization in Transformers V5: Simpler, Clearer, and More Modular

Companies Mentioned

Why It Matters

Key Takeaways

Pulse Analysis

Ask Pulse AI:

Tokenization in Transformers v5: Simpler, Clearer, and More Modular

Transformers v5 redesigns how tokenizers work

Table of Contents

What is tokenization?

The tokenization pipeline

Tokenization algorithms

Accessing tokenizers through transformers

The tokenizer class hierarchy in transformers

PreTrainedTokenizerBase – common interface for all tokenizers

TokenizersBackend – wraps the Rust tokenizers library

PythonBackend – pure‑Python mixin

SentencePieceBackend – handles SentencePiece models

AutoTokenizer automatically selects the correct tokenizer class

v5 separates tokenizer architecture from trained vocab

The problem with v4

The v5 solution

Example: training a LLaMA‑style tokenizer from scratch

Summary

Further resources

Comments

Accessing tokenizers through `transformers`

The tokenizer class hierarchy in `transformers`

`PreTrainedTokenizerBase` – common interface for all tokenizers

`TokenizersBackend` – wraps the Rust `tokenizers` library

`PythonBackend` – pure‑Python mixin

`SentencePieceBackend` – handles SentencePiece models

`AutoTokenizer` automatically selects the correct tokenizer class

Accessing tokenizers through `transformers`

The tokenizer class hierarchy in `transformers`

`PreTrainedTokenizerBase` – common interface for all tokenizers

`TokenizersBackend` – wraps the Rust `tokenizers` library

`PythonBackend` – pure‑Python mixin

`SentencePieceBackend` – handles SentencePiece models

`AutoTokenizer` automatically selects the correct tokenizer class