Block: The Transformer · Course topic: AI
Focus areas: Where does the Transformer get the order of words from? Why is Self-Attention alone not sufficient? How is order represented mathematically?
The central problem we examine in this lecture is the following: Self-Attention, as introduced in the Transformer, processes all tokens of an input sequence simultaneously and is, in its pure form, invariant to permutations of the input. This means that if the tokens of a sentence are arranged in any order, then — without additional information — the attention mechanism alone cannot distinguish their original positions. This observation was explicitly noted and addressed in the original Transformer paper, where the authors introduced an explicit representation of each token's position (Vaswani et al., 2017).
An illustrative analogy: Imagine a box of colored marbles, each color representing a word. Self-Attention only sees the colors and their mutual relationships, but not the order in which the marbles were drawn from the box. Position encoding adds a mark to each marble — turning a mere set of colors into an ordered chain.
In practice, position information is added to the word vectors (embeddings) before the attention layers operate. You can think of it like this: each token vector receives a small "time" or "location" component indicating its position in the sequence. This allows query and key vectors in the attention computation to use ordering information to form weighted connections between specific positions. The implementation choice for how to encode this position information is not uniquely fixed; different variants have been established in research.
A short linguistic example suffices to illustrate: The sequence "Der Hund beißt den Mann" (The dog bites the man) is semantically different from "Der Mann beißt den Hund" (The man bites the dog). Self-Attention would need to know the order to determine which subject relates to which verb. Without position information the same tokens in both sentences cannot be distinguished.
For further illustrative elaborations and visualizations I refer to explanatory online teaching materials ("The Illustrated Transformer" and "The Annotated Transformer"), which explain the structure and practical use of position encodings (Alammar, 2018; Harvard NLP, 2018). The underlying design principle — that positions must be provided in addition to token embeddings — goes back directly to the original Transformer publication (Vaswani et al., 2017).
Mathematically, Transformer layers operate with vector representations. For a sequence of T tokens there are input embeddings x₁, x₂, …, x_T (each a vector of dimension d_model). Position encoding provides for each index pos (1 ≤ pos ≤ T) a vector p(pos) of the same dimensionality. In the classical variant of the original Transformer paper the input representations are combined by addition before the first attention layer: z_pos = x_pos + p(pos). This addition makes the position information directly available in the subsequent computations of queries, keys and values (Vaswani et al., 2017).
Vaswani et al. propose two main variants: fixed sinusoidal functions and learnable position embeddings. The sinusoidal form given in the paper is defined as follows (index i runs over the components of the d_model-dimensional vector):
For the even components 2i: p(pos)_{2i} = sin(pos / 10000^{2i/d_model}). For the odd components 2i+1: p(pos)_{2i+1} = cos(pos / 10000^{2i/d_model}). This form generates for each position a vector pattern of sine and cosine waves with differently scaled frequencies. Vaswani et al. motivate this choice by noting that such basis functions represent relative positions in a way that allows subsequent layers to respond to (relative) shifts in position via linear combinations; moreover, sin/cos functions provide a continuous representation that is in principle extrapolatable to longer sequences (Vaswani et al., 2017).
As an alternative to fixed functions, position embeddings can also be modeled as learnable parameters, similar to word embeddings. This approach is used, for example, in BERT: there vectors for each possible position index are learned as parameters and trained together with token embeddings (Devlin et al., 2018). A practical difference is that learnable embeddings often show better performance within the lengths seen during training, but may generalize worse to significantly longer sequences because no explicit analytic extrapolation is provided (Devlin et al., 2018). The exact performance differences, however, depend on architecture, data and task; there is no universally valid verdict.
Besides these absolute position representations, a third important category exists: relative position representations. Shaw, Uszkoreit and Vaswani (2018) propose to introduce the relative relation between query and key positions directly into the attention logits. Practically, additional terms are used in the attention computation that represent the relative distance (j − i) between token i and j. Such representations can be advantageous for tasks where relative distances are more important than absolute indices (for example many translation tasks or dependency analyses), because they directly incorporate distance information between token pairs (Shaw et al., 2018).
Formally, position information influences the attention weights ω_{ij} via the standard computation:
ω_{ij} = softmax_j( (Q_i · K_j) / sqrt(d_k) + b_{ij} ), where b_{ij} can be an additional term that depends on relative or absolute position. In the original Transformer without a special relative term b_{ij} arises implicitly from the summation x_j + p(j) and the resulting K_j; in variants with relative encodings b_{ij} is modeled explicitly (Shaw et al., 2018).
Important implementation details: Codebases and teaching materials often show the simple variant of adding embeddings with position vectors before computing the linear projections for Q, K, V. This practical approach is presented and explained in tutorial implementations and commented implementations (Harvard NLP, 2018; Alammar, 2018).
In conclusion, the community has no single binding standard for position encoding; fixed sin/cos, learnable absolute embeddings and various relative methods are in use and each has conceptual pros and cons. Which approach is preferable in a given case depends on the application, available compute and the length distributions seen in training data. This is an active research area with ongoing comparisons and modifications (Vaswani et al., 2017; Devlin et al., 2018; Shaw et al., 2018).
Position encoding is now found in practically all Transformer-based systems for language, translation, text generation and many other sequence tasks. The concrete choice of position representation can have performance effects: for example, in models trained on fixed input lengths a learnable position embedding is often effective; in applications that expect longer sequences than seen in training, sinusoidal or specially constructed position embeddings can extrapolate better (Vaswani et al., 2017; Devlin et al., 2018).
Limitations and open questions: There is no universal solution that is optimal in all settings. Relative position representations are more powerful in some tasks but increase implementation complexity. Scalability to very long sequences is also an active research topic; many extensions and alternatives investigate how positions can be represented efficiently and robustly for thousands or millions of tokens. The literature shows ongoing developments here, so the assessment of individual methods depends on the latest research (Shaw et al., 2018; Vaswani et al., 2017).
To consolidate what you have learned I suggest three small thought and homework exercises that can also be investigated practically with existing implementations:
1) Take two sentences with identical tokens but different order (e.g. "Der Hund beißt den Mann" and "Der Mann beißt den Hund") and examine the attention weights in a Transformer encoder block. Observe how the distributions change if you remove position encodings or replace them with random ones. This exercise makes the role of positions in the weighting process visible (Harvard NLP, 2018).
2) Implement a short experiment: replace the sinusoidal encodings in a small Transformer with learnable position vectors, train on a limited dataset and compare validation error as well as behavior on inputs longer than those seen during training. Discuss whether and how generalization to longer sequences breaks down (Vaswani et al., 2017; Devlin et al., 2018).
3) Read the paper by Shaw et al. (2018) and explain how an explicit relative term is integrated into the attention logits. Consider for which tasks relative distances are more sensible than absolute indices and why (Shaw et al., 2018).
In closing: The necessity to encode order is conceptually undisputed in the Transformer architecture; however, the methods to represent it are diverse and subject to active research. For practical work with Transformers it is important to consciously choose the variant and consider its implications for generalization and computational cost.