Part 1 (approx. 20 minutes): Basic understanding — Why not work directly with words? What is a token?
When we humans read or speak, we usually think in words, stems, or meanings. Computer programs, and thus neural language models, however, see character strings — sequences of bytes or Unicode characters. Mapping every possible word directly to its own representation is impractical for practical reasons: the number of possible word forms (inflections, compounds, proper names, typos, rare technical terms, borrowings) is very large and depends on the language. Therefore, methods have been established that split text into smaller, reusable units: tokens.
In this context, a "token" denotes a unit that a tokenizer produces from raw text and that the model uses as input. Tokens are not a universal linguistic category like "word" in the traditional sense; they are algorithmically defined units that can represent sequences of letters, parts of words, whole words, or punctuation marks. Common tokenizations for modern transformer models operate at the level of subwords or byte-like units, not primarily at the level of semantic words (see methodological descriptions of subword tokenizers and tokenizer implementations).
A useful analogy is building with Lego bricks. Words are like finished models (e.g., a car), tokens are the individual bricks. If you allowed only finished models as building blocks, you would need infinitely many different models to cover all possible combinations. With a manageable set of bricks (tokens), however, you can assemble a very large number of words and word combinations.
This approach has practical advantages: it reduces the required vocabulary size, allows modeling of rare or new words by combining existing parts, and reduces problems with "out-of-vocabulary" (OOV) words, i.e., words the model does not know at all. The idea of using subword units instead of complete words is described in several established works and implementations (see in particular works on subword methods and practical tokenizer implementations).
(See Sennrich et al., 2015; Kudo & Richardson, 2018; Devlin et al., 2018; Hugging Face documentation) [see bibliography]
Part 2 (approx. 20 minutes): Deepening and technical terms
The following introduces the most important technical terms and foundations systematically and explains how they relate to each other.
Token and tokenizer
A tokenizer is an algorithm that splits incoming text into tokens. There are different principles for tokenizers. Three common families are: word-based (very rare for modern models), subword-based (such as Byte-Pair Encoding, WordPiece), and byte- or byte-like based methods (like some variants of SentencePiece, which also provide byte- and UTF-8-resilience). Subword methods choose a finite set of subword symbols (a vocabulary) and split text so that frequent word parts appear as single tokens, while rarer words are split into multiple tokens.
Byte Pair Encoding (BPE) and its successors
Byte Pair Encoding (BPE) is a simple data-driven method that repeatedly merges frequent pairs of characters or symbols into new entries in the vocabulary. This produces subword units that often represent common prefixes, suffixes, or stems. BPE was adapted for machine translation to better handle rare words (Sennrich et al., 2015). WordPiece, a related method, uses a probabilistic criterion and was used in large language models like BERT (Devlin et al., 2018). SentencePiece is an implementation that can operate independently of whitespace and supports both BPE and unigram-based models (Kudo & Richardson, 2018).
Why words can consist of a different number of tokens
Whether a word is split into one, two, or multiple tokens depends on the vocabulary used and the tokenizer algorithm. Frequent words or word parts are typically learned as single tokens. Rare words, long compound words, or creative spellings are split into several known subpieces. For example: in a vocabulary "Haus" might appear as a single token, while "Hausaufgabe" could be "Haus" + "aufgabe" or "Haus" + "##auf" + "gabe", depending on tokenizer conventions (WordPiece often uses special continuation markers). Some tokenizers operate at the character or byte level, so practically every character or byte can be a token; others prefer longer subword units. The exact split is therefore a consequence of the tokenizer's training data and the chosen algorithm.
Consequences for computational effort and model architecture
Transformer models process sequences of tokens. The computational costs (especially for self-attention) depend on the length of that sequence. In the original Transformer description, the computational complexity of self-attention with sequence length n is given as O(n^2 · d), where d is the model dimension (Vaswani et al., 2017). This means: if the number of tokens doubles, the cost for attention roughly increases by a factor of four (all other parameters equal). Accordingly, the choice of token granularity directly affects training time, inference time, and thus costs in cloud environments.
From this follows an important trade-off: coarser tokens (e.g., a larger subword vocabulary and thus on average fewer tokens per word) reduce sequence length, but can increase the vocabulary size and memory requirements (e.g., the embedding matrix). Finer-grained tokens (e.g., byte-level) lead to longer sequences and thus higher computational cost in attention, but they enable very robust handling of arbitrary character sequences and a smaller number of unknown characters. Research on "Efficient Transformers" aims, among other things, to mitigate the quadratic scaling problem, but also shows that sequence length remains a central cost factor (Tay et al., 2020).
Token counting in practical systems and billing
In practical APIs and usage models, billing is often done per token or token limits determine the maximum usable context length. Providers document how they count tokens and which tokenizers they use; this therefore directly affects usage costs and limits when deploying models in production systems (see platform documentation and tokenizer demos). It is important to know that the "token" in billing is the same as what the model tokenizer produces; different models can use different tokenizers, so the same text input can result in a different number of tokens across systems.
(See Vaswani et al., 2017; Tay et al., 2020; OpenAI Tokenizer documentation; Hugging Face Tokenizers) [see bibliography]
Transparency and uncertainties
There is consensus that subword tokenization is a practical and now widespread solution to the problem of huge word vocabularies. Precise effects — for example how strongly a particular tokenization affects model performance or costs in a particular language — however depend heavily on the training data, language structure, and implementation. Quantitative statements about average token lengths or costs are therefore data- and system-specific; general trends (e.g., that more tokens usually mean more computation) are, however, robust.
Part 3 (approx. 10 minutes): Applications, limits and thought exercises
Finally, we look at concrete applications, practical limitations and small thought exercises to test understanding.
Applications and practical implications
In applications such as machine translation, text generation, question-answering systems, or autocomplete, tokenization determines two fundamental properties: first, robustness to unknown or rare words (subword tokenizers help a lot here). Second, efficiency: longer token sequences require more computation time and more memory for the same input, especially due to the quadratic cost in self-attention. When deploying in production systems, this means one must deliberately choose tokenizer, vocabulary size and context length to achieve an acceptable balance of accuracy versus runtime/cost. Tokenizer implementation and platform documentation often provide practical recommendations and tools to measure and optimize token usage.
Limits
Tokenization is not a panacea. It does not automatically resolve semantic ambiguities, ironic turns, or complex pragmatics. Also, some languages are harder to handle due to their morphology or orthography; very agglutinative or polysynthetic languages may require special considerations in tokenization. Finally, the effects are language- and domain-specific: a tokenizer that works well for general English is not necessarily optimal for medical texts or code-mixed content.
Small thought exercises for deepening
1. Take a sentence in your native language and tokenize it with two different freely available tokenizers (e.g., a byte-level tokenizer and a subword tokenizer). Observe how many tokens are produced and which word parts are separated. Reflect on which tokenization seems more advantageous for your application (e.g., translation vs. autocomplete) and why. 2. Imagine a particular word in your data occurs extremely often. Discuss how that should influence the decision about the vocabulary and what consequences it can have for storage/embedding tables. 3. Consider the pros and cons of coarser tokenization (fewer, longer tokens) versus finer tokenization (more, shorter tokens) for training a large model — both in terms of model size and inference costs.
These exercises are intentionally general: concrete, quantitative analyses each require access to the training data, the tokenizer implementation and measurements of runtime/costs in the respective infrastructure.