Lecture 13 — Embeddings: How are words translated into numbers?

Introduction

In this lecture we clarify what "embeddings" are, how words become dense numeric vectors, why similar terms are close together in the vector space, and why embeddings are more than simple dictionaries. The presentation is based on the scientific consensus from research on distributional semantics and on established technical publications (including work on Word2Vec, GloVe, Transformer- and BERT-models). Where uncertainties or open research questions exist, I explicitly point them out.

Part 1 (20 minutes) — Basic idea, simple analogies and examples

Time: ~20 min

Embeddings are numerical representations (vectors) of words or other language units. The central idea: assign each word a point in a multidimensional space so that geometric relationships between points reflect linguistic relationships. This idea builds on the so-called distributional hypothesis: words that occur in similar contexts have similar meanings (this principle is the starting point of many works on word vectors, cf. for example [Mikolov et al., 2013] and [Pennington et al., 2014]).

Analogy 1 — Map: Imagine you have a map where cities are arranged by similarity rather than by geographic location. Cities with similar climate or culture would lie close to each other. In embeddings the word is the "place" and the "neighborhood" corresponds to similar usage in language.

Analogy 2 — Color space: Colors are often represented as points in a three-dimensional space (e.g., RGB); similar hues lie close together. Similarly, in an embedding space semantically close words (e.g., "dog", "cat", "pet") lie near each other.

Concrete example: In early embedding models a striking property appeared: simple vector operations often correspond to linguistic relations. The famous example is roughly: Vector("King") − Vector("Man") + Vector("Woman") ≈ Vector("Queen") (see [Mikolov et al., 2013]). This example illustrates that relationships between words can be represented as directions in the space.

How do these points arise? In short: through a learning process based on large amounts of text that aims to predict contexts or to explain statistical relationships of word pairs. Models like Word2Vec learn, for example, which words are likely in the neighborhood of a target word. Other methods like GloVe build on global co‑occurrences (how often two words occur together) and summarize this information into vectors (cf. [Mikolov et al., 2013], [Pennington et al., 2014]).

Important point: Embeddings are not just "dictionary entries" (a list of attributes). They are dense, continuous representations that encode semantic and partly syntactic information in the geometry of the space.

Part 2 (20 minutes) — Deeper dive: technical terms and mechanisms

Time: ~20 min

We now introduce central technical terms and explain how the most common methods work — without formal derivation, but precise enough to understand the mechanics.

Distributional Hypothesis

The distributional hypothesis states that similarity of meaning is reflected by similarity in usage (context). Modern embedding methods operationalize this by using statistics about context-word pairs to optimize vectors. The idea appears as a basic assumption in many works on word vectors (e.g. [Mikolov et al., 2013]; [Pennington et al., 2014]).

Static vs. contextual embeddings

- Static embeddings: Each word has exactly one vector regardless of context. Examples: Word2Vec (skip-gram, CBOW) and GloVe. Such vectors summarize the average usage of a word across the training corpus (see [Mikolov et al., 2013]; [Pennington et al., 2014]).
- Contextual embeddings: A word is represented differently depending on the sentence or environment. They are produced by context-dependent models, especially Transformers, such as BERT. The token "Bank" in "bench" vs. "bank" (financial institution) receives different vectors because the model considers the surrounding context (cf. [Devlin et al., 2018]; [Vaswani et al., 2017]).

Basic principles of training objectives

Two common ways to obtain embeddings:

1) Prediction-based methods: Models learn to predict surrounding context words for a target word (or vice versa). Word2Vec is a representative of this class. The training objective is to maximize probabilities so that true context words are more likely than random negatives (cf. [Mikolov et al., 2013]). Negative sampling is a technique that helps compute this optimization efficiently.

2) Matrix factorization / global methods: Here statistical co‑occurrence counts (how often word i occurs together with word j) are arranged in a matrix and this matrix is factorized so that dense vectors emerge which explain the main relationships. GloVe follows this approach by defining a loss that requires reconstruction of log co‑occurrences (cf. [Pennington et al., 2014]).

Geometry and similarity measures

The spatial closeness of two word vectors is usually measured with the cosine metric (cosine similarity): it measures the angle between two vectors and is robust to differing normalizations. Small angles (high cosine) mean high semantic similarity. This metric is practically used to determine nearest neighbors in the space (see [Mikolov et al., 2013]).

Why simple algebraic relations appear

That relations like "King − Man + Woman ≈ Queen" appear can plausibly be attributed to the fact that training objectives encourage consistent patterns between word pairs. If certain relationships occur regularly in the corpus (e.g., gender markers, tense forms), they manifest as directions or subspaces in the vector space. The exact mechanisms are an active research topic; however, there is broad agreement that the training objectives and statistical properties of correlated word pairs favor these linear structures (cf. [Mikolov et al., 2013]; [Pennington et al., 2014]).

Contextual embeddings and Transformer

Transformer models produce token representations via stacked self-attention and feed-forward networks; these representations are context-dependent and are created by pretraining on large amounts of text. BERT, for example, uses the Masked Language Model (MLM) as a pretraining objective: parts of a sentence are masked and the model learns to predict the masked tokens from the context. This produces embeddings that capture semantic nuances and word sense in the given context (cf. [Vaswani et al., 2017]; [Devlin et al., 2018]). For a summarizing presentation of findings about how BERT works I refer to a recent survey (A Primer in BERTology; cf. [Rogers et al., 2020]).

Dimensions, normalization, pretraining and fine-tuning

Practical decisions when creating embeddings concern vector size (typically between a few 50 and several 1,000 dimensions), normalization procedures, and whether to use pretrained embeddings or to train your own. Pretrained embeddings (static or contextual) are often fine-tuned in downstream tasks to achieve domain-specific performance gains (see [Devlin et al., 2018]; [Reimers & Gurevych, 2019] for sentence/applications).

Open questions and uncertainties

Although many surprising properties of embeddings are empirically established, questions remain about the full interpretation of the geometry and robustness across different corpora. Research indicates that some observed regularities are model- and data-specific and do not always hold universally (cf. [Rogers et al., 2020]).

Part 3 (10 minutes) — Applications, limitations and thought exercises

Time: ~10 min

Applications (concrete and substantiated)

Embeddings are used in many practical areas:

- Semantic search: Documents are described by embeddings; queries are mapped to similar embeddings to find semantically relevant hits (see e.g. approaches with Sentence-BERT for sentences and documents; [Reimers & Gurevych, 2019]).
- Text classification and information extraction: Embeddings serve as input vectors for classifiers or further models (general practice described among others in [Devlin et al., 2018]).
- Clustering and exploration: Topics or lexical fields can be visualized and analyzed by clustering in the embedding space (concrete use in NLP pipelines, cf. [Pennington et al., 2014]).
- Machine translation and other NLP tasks: Embeddings are an integral part of modern language models that provide better representations in many applications (cf. [Vaswani et al., 2017]).

Limitations and risks (scientifically documented)

- Polysemy: Static embeddings conflate different meanings of a word into one vector and cannot separate context-specific nuances; this problem is mitigated by contextual embeddings but not completely eliminated (cf. [Devlin et al., 2018]).
- Biases: Pretrained embeddings reflect systematic biases in the training corpus; empirical work shows that gender and other stereotypes can be encoded in word vectors (see [Bolukbasi et al., 2016]). This is a practical and ethical problem in applications.
- Interpretability: Individual dimensions are generally not directly interpretable; meaning resides in relations and patterns, not in isolated components (discussion in [Rogers et al., 2020]).
- Data dependence: Quality and properties of embeddings strongly depend on the training corpus (language, domain, genre).

Small thought exercises for practice

1) Nearest-neighbour experiment: Choose three different words ("Bank", "apple", "run") and consider which neighbors a static embedding model vs. a contextual model would likely provide. Discuss why the neighborhoods may differ (relation: polysemy, [Mikolov et al., 2013]; [Devlin et al., 2018]).
2) Analogy probe: Come up with an analogy similar to "King − Man + Woman". Which conditions in the corpus would need to be met for such an analogy to be reliably represented? (Hint: regular, consistent patterns and sufficient data; [Mikolov et al., 2013]).
3) Bias analysis: Choose a cultural or occupational lexical field and examine which associations embeddings in your data material show. What consequences could this have for an application? (See empirical studies on biases, [Bolukbasi et al., 2016].)

In conclusion: Embeddings are a central tool of modern language processing. They link statistical patterns in text with geometric structures that make semantic relationships visible. At the same time their limitations — especially regarding bias, interpretability and data dependence — are well documented and must be considered in any application.