Skip to main content

Byte Pair Encoding (BPE) Tokenization

Byte Pair Encoding (BPE) is the standard subword tokenization method used by most large language models. Originally developed as a data compression algorithm in the 1990s, it is now fundamental to how LLMs process text across languages.

How BPE Works​

BPE starts with a base vocabulary of individual characters (or bytes) and iteratively builds larger tokens:

  1. Initialize: Start with individual characters or bytes as the base vocabulary
  2. Find frequent pairs: Scan training data to find the most frequently occurring pair of adjacent symbols
  3. Merge: Combine the most frequent pair into a single new token
  4. Repeat: Continue merging until reaching a predetermined target vocabulary size (typically 32,000 to 256,000 tokens)

This ensures common words remain whole, while rare words are broken down into logical, recognizable chunks.

The Vocabulary Size Crisis​

If an AI tried to memorize every single whole word in English, its vocabulary would need hundreds of thousands of words. Adding Spanish, Mandarin, Arabic, Hindi, and Russian would explode the vocabulary into millions of words, making the model's embedding layer so massive it would consume all available GPU memory before even processing sentences.

BPE solves this by creating a highly efficient, shared multilingual alphabet.

Cross-Lingual Subword Sharing​

Many languages share common roots, prefixes, and suffixes. Because BPE breaks words down into subwords, different languages can reuse the exact same tokens.

The Romance Language Advantage: Words like "traduction" (French), "traducción" (Spanish), and "traduzione" (Italian) share the root token traduc. BPE keeps traduc as a single token in its vocabulary, which all three languages share.

Loanwords and Technical Terms: Global terms like "Internet," "Python," or "AI" are represented by the same tokens across dozens of languages.

Byte-Level BPE (BBPE)​

Early tokenizers struggled with languages that don't use the Latin alphabet (like Chinese, Japanese, or Arabic), often resulting in "unknown word" errors (<UNK>).

Modern LLMs use Byte-Level BPE (BBPE), which merges bytes (the raw 0s and 1s representing text via UTF-8 encoding) instead of characters.

Character vs. Byte Comparison​

ApproachVocabulary RequirementHandling of Non-Latin Scripts
Character-Based BPENeeds thousands of unique characters to cover alphabets, Kanji, Cyrillic, Arabic, etc.High risk of running into unseen characters (<UNK>).
Byte-Level BPE (BBPE)Needs a base vocabulary of only 256 fundamental bytes.Zero unknown characters. Every language on Earth can be expressed using combinations of those 256 bytes.

Because the base alphabet is just 256 bytes, the model can allocate the rest of its vocabulary budget to the most common byte combinations across all languages.

Token Fertility and the Multilingual Tax​

While BPE allows an LLM to understand every language, it doesn't treat them all equally. This is measured by token fertility—the average number of tokens it takes to represent a single word.

Token Fertility Example:

The phrase "Higher Education" is 2 words.

  • English: Higher + Education (2 tokens)
  • Telugu: ఉన్నత + విద్యా + భ్యాసం (3 words, but breaks into 8+ tokens)

Because training data is predominantly English, BPE optimizes heavily for English patterns. English words usually equal 1 token. Non-Latin scripts (like Hindi, Korean, or Arabic) get broken down into much smaller byte-chunks, meaning a single word might cost 4 or 5 tokens.

This is why non-English text often costs more and processes slightly slower in commercial LLM APIs.

Advantages of BPE​

  1. Handles rare words: Breaks unknown words into known subword units (e.g., "cryptocurrency" → "crypto" + "currency")
  2. Memory efficient: Fixed vocabulary size regardless of language diversity
  3. No unknown tokens: Byte-level variant guarantees zero <UNK> errors
  4. Cross-lingual transfer: Shared subwords enable better multilingual understanding
  5. Balances granularity: Avoids extremes of character-level (too granular) and word-level (too rigid) tokenization

Other Tokenization Methods​

  • WordPiece: Similar to BPE but uses likelihood-based scoring (used by BERT)
  • Unigram: Probabilistic subword tokenization (used by some models)
  • SentencePiece: Language-agnostic tokenizer treating text as raw byte streams