Lecture 16: Multi-Head Attention

Part 1 (approx. 20 minutes): Understandable Introduction with Analogies and Examples

Goal of this section:

I explain why modern transformer models do not use just a single attention function, but run several so-called attention heads in parallel. I use easy-to-understand analogies and concrete sentence examples to make the principle and its effect on language understanding comprehensible. The technical basis (scaled dot-product attention, linear projections) is mentioned only as far as necessary for understanding the idea; for formal details I refer to the original description of the transformer model (Vaswani et al., 2017).

Imagine you are listening to an orchestra. Each instrument contributes information: the violin may carry the melody, the cello provides the foundation, and the drums give rhythm. If you listen to only one instrument, you miss much of the overall sound. Similarly, multi-head attention does not consider the input sequence just once, but from several "perspectives" simultaneously. Each attention head is like an individual instrument: it focuses on certain kinds of relationships or patterns in a sentence.

Consider a concrete example: "The cat that the dog chased was black." To understand the word relationships correctly, we must accomplish at least two tasks: the syntactic relation (which subject belongs to which verb) and the reference resolution (which noun "the" or "the cat" refers to in context). A single attention mechanism could theoretically cover both tasks, but multiple heads can handle these tasks in parallel and in a specialized way: one head learns typical subject–verb connections, another head focuses on embeddings and appositions, a third on pronoun resolution. This produces a richer and more robust representation of the sentence.

In short: multi-head attention makes it possible to capture different kinds of relationships in parallel, specialized "channels." This parallel work is especially useful for natural language, because language contains many simultaneous relations (syntax, semantics, anaphora, colloquialisms, idioms). Practice has shown that this architecture has contributed substantially to the strong performance of transformers on language tasks (Vaswani et al., 2017).

Part 2 (approx. 20 minutes): Deepening and Introduction of Key Technical Terms

Goal of this section:

I introduce the central technical terms, explain formally what each attention head does, and present empirical findings on the functionality of heads from research.

Formally, each attention head works with its own linear projections of the input vectors. At the core, an attention computation consists of three elements, known in the literature as Query (Q), Key (K) and Value (V). The scaled dot-product attention computes the compatibility between query and key, scales the result and applies a softmax function to form weights over the values. Briefly noted: Attention(Q,K,V) = softmax(Q K^T / sqrt(d_k)) V. In the multi-head mechanism these Q/K/V projections are executed multiple times — once per head with their own weight matrices — the resulting head outputs are then concatenated and linearly transformed again, so that the model can examine different subspaces of the representations simultaneously (Vaswani et al., 2017).

The key concept is specialization through different projections: each head sees the input from a different "linear viewpoint." Practically, this means one head recognizes patterns that appear as linear relationships in a particular projection subspace, while another head filters out different patterns. This property allows the model to model multiple types of dependencies in parallel and efficiently.

What kinds of patterns do attention heads show in practice? Empirical analyses and visualizations have produced several recurring observations. A systematic finding is that some heads take on clear linguistic roles: some heads show strong focus on syntactic relations such as subject–verb or object–verb; other heads correlate with word repetition, prepositions or punctuation. Such observations have been documented in analyses of pretrained transformers and made accessible with visualization tools (Clark et al., 2019; Vig, 2019). At the same time, investigation results show that not all heads have a clearly interpretable function; many heads are redundant or spread their activity diffusely across tokens (Michel et al., 2019).

An important experimental result in the research is: there are both highly specialized and redundant heads. Michel and colleagues have shown that in some models many heads can be removed without the overall performance dropping significantly. This does not imply that the remaining heads are arbitrarily interchangeable, but that the model contains both specialized and partially redundant mechanisms. This result is relevant for model compression and for understanding how learning capacity is organized in the transformer (Michel et al., 2019).

It is also important to weigh interpretations carefully: visualizations of attention weights provide hints about possible functions of individual heads, but they are not proof that a head is causally responsible for a particular prediction. Interpretability research discusses how reliable attention weights are as an explanatory tool. Visualization tools help form and test hypotheses, but they do not replace controlled intervention or ablation analyses (Vig, 2019; Vaswani et al., 2017).

Part 3 (approx. 10 minutes): Applications, Limitations and Thought Exercises

Goal of this section:

I show how multi-head attention acts in real applications, name limits of current knowledge and propose small thought exercises to consolidate what has been learned.

In concrete applications, transformer models with multi-head attention show clear advantages. In machine translation, multiple heads help represent different word and phrase alignments in parallel; in question answering some heads can highlight relevant context positions while others secure syntactic relations; in text summarization and dialogue systems the different heads enable capturing information hierarchies and discourse relations. These practical successes stem from the combined ability to model multiple relationships simultaneously, producing richer representations in higher layers (Vaswani et al., 2017; Clark et al., 2019).

At the same time there are limits and open questions. First: not every head is easily interpretable; interpretations often rely on correlations in visualizations and must be experimentally verified. Second: research shows redundancies, raising questions about efficiency and minimal architectures; the exact contribution of individual heads to overall performance is the subject of ongoing studies (Michel et al., 2019). Third: the observed linguistic patterns are not necessarily universal across all models and tasks; they can depend on architecture size, training data and optimization details (Clark et al., 2019).

To deepen understanding, three short thought exercises:

1) Take the sentence "The man whom the woman saw waved." Which two kinds of relationships would you expect different attention heads to cover so that a model correctly captures the sentence structure? Think about the coupling of relative-clause references and the main-clause structure.

2) Consider the word "bank" in "He is sitting on the bank" versus "He works at the bank." What role could multi-head attention play in resolving polysemy, and how would different heads be involved?

3) Design an experiment: systematically remove individual heads in a pretrained transformer layer and observe the change in performance on a downstream task (e.g., Named Entity Recognition). What would large performance drops versus small effects tell you about the roles of the removed heads? (Note: Michel et al., 2019 performed such head-pruning analyses and report both critical and redundant heads.)

In conclusion: multi-head attention is not an end in itself, but a design choice that allows transformers to model multiple relationships simultaneously and thereby produce richer contexts for language processing. Interpretations of individual heads are useful, but part of an ongoing research process into the robustness, efficiency and explainability of these mechanisms.