What is tokenization in LLMs?

Tokenization in Large Language Models (LLMs) is the process of breaking down text into smaller units called tokens, which the model can understand and process. A token can represent a word, part of a word, a single character, or even a punctuation mark. For example, the sentence “ChatGPT is useful!” might be split into tokens like ["Chat", "G", "PT", " is", " useful", “!”]. This step is critical because LLMs don’t interpret raw text directly—they work with numerical representations of tokens. Tokenization bridges human-readable text and the numerical data structures (like vectors) that models use for computation. The exact rules for splitting text into tokens depend on the tokenization algorithm and the model’s training data.

Most LLMs use subword tokenization methods like Byte-Pair Encoding (BPE) or WordPiece. These algorithms balance between treating whole words as single tokens and splitting rare or complex words into smaller parts. For instance, BPE starts by splitting text into individual characters and then iteratively merges the most frequent pairs of existing tokens. This allows the model to handle unknown words by breaking them into known subword units. For example, the word “unhappy” could be split into ["un", “happy”], where both subwords might already exist in the model’s vocabulary. The size of the tokenizer’s vocabulary—often ranging from tens of thousands to over 100,000 tokens—is a key design choice. A larger vocabulary can represent common words more efficiently but may struggle with rare terms, while a smaller vocabulary increases the likelihood of splitting words into subunits.

Tokenization impacts how LLMs handle specific tasks and languages. For instance, models might split “don’t” into ["do", “n’t”], which preserves grammatical meaning but requires the model to learn relationships between such splits. In multilingual contexts, tokenization becomes more complex because scripts (like Chinese vs. Latin) and word structures vary widely. A poorly designed tokenizer can lead to inefficiencies, such as excessive token counts for languages with no spaces between words (e.g., Japanese). Developers should also be aware that token limits—like GPT-4’s 128k-token context window—are based on tokens, not raw text, so input length depends heavily on the tokenizer’s behavior. Understanding tokenization helps in debugging issues like unexpected model outputs or errors when processing unusual text patterns, such as code snippets or non-standard punctuation.