Lecture 17 — Pretraining

Part 1 (approx. 20 minutes): Basic Understanding — How Pretraining Works (Illustrative)

At the core of pretraining a large autoregressive language model like GPT is a single, clear learning task: the model is shown a very large amount of text and it learns to predict the next token for any sequence of words or tokens as accurately as possible. This task is formally expressed as maximizing the conditional probability of the next token given the preceding tokens; in practice this is achieved via optimization (gradient descent) of a probabilistic model, as described in the publications on GPT-2 and GPT-3 (Radford et al., 2019; Brown et al., 2020).

A memorable analogy: Imagine a child who reads millions of books and repeatedly tries to guess the next word in a sentence. The more it reads and the more varied text types it sees, the better it becomes at recognizing typical continuations — for grammar, common phrases, narrative patterns, and also facts that frequently appear in that form. A statistical model learns in the same way: it recognizes probability patterns in the data, not "meaning" in the philosophical sense, but representations and rules that recur in the data.

Why is the simple task of "next-token prediction" so powerful? Because natural language is highly structured: forms of syntax, semantic regularities, coreference and often world knowledge appear as recurring patterns across enormous amounts of text. Studies and reviews show that models trained on this task develop internal representations that encode information about syntax and semantics (Rogers et al., 2020; Marvin & Linzen, 2018). They do not learn abstract rules explicitly, but statistically robust patterns that prove useful for many tasks.

Important note on the data foundation: In practice researchers and developers use large, heterogeneous datasets (Common Crawl, web scrapes, books, Wikipedia, etc.), as documented in the descriptions of GPT-2 and GPT-3. Thus the models "see" many different writing styles and often also erroneous, biased or outdated information. These data characteristics shape what the models learn.

Part 2 (approx. 20 minutes): Technical Deepening — Terms and Mechanics

Starting point and objective: The training task is formally framed as minimizing the negative log-likelihood (or equivalently cross-entropy) of the actually following tokens under the model. For a sequence x1…xT the objective summed over t is: −log p(x_t | x_1…x_{t-1}). In practice this means the model produces for each subsequence a probability distribution over the next token (the output of a softmax), and the parameters are adjusted so that the probability of the actually occurring continuation increases. This approach is explained in the major publications on transformer-based language models (Brown et al., 2020; Radford et al., 2019).

Architectural connection: The large models commonly used today for this task are based on the Transformer architecture, which uses self-attention to model dependencies within the input sequence; the original description can be found in Vaswani et al., 2017. The concrete learning work — gradient calculation and weight updates — uses the same procedures (backpropagation, stochastic optimizers) covered earlier in the lecture series. The model's "memory" emerges here in the form of learned weights that generalize patterns from the examples.

Why prediction ⇒ language understanding (not metaphysical, but functional): Multiple lines of research show that the prediction task produces representations useful for many downstream tasks. Concretely, targeted tests show that models can predict syntactic dependencies and partly semantic relations even though they were never directly trained on those tasks (Marvin & Linzen, 2018; Rogers et al., 2020). Further empirical work on scaling and generalization shows that with larger model and data sizes capabilities emerge that were not visible in smaller models (Kaplan et al., 2020; Wei et al., 2022). These findings support the scientific consensus that next-token training is a powerful method to learn broadly useful linguistic representations.

Limits of the learning signal and what it does not guarantee: The prediction task is purely statistical. The model optimizes prediction accuracy on the training data distribution; this does not automatically imply that the model possesses reliable, causal, or fact-based knowledge, nor that it will provide responsible, conflict-free answers. Problems such as hallucinations (inventing facts), reproduction of biases from training data, and lack of robustness to distribution shift are documented in the literature (Lin et al., 2022; Bender et al., 2021). Methods like fine-tuning and human feedback (e.g., InstructGPT/training with human feedback) are used to improve behavior after pretraining, but they do not address all structural limits of the pretraining objective (Ouyang et al., 2022).

Scaling and effects: Work on scaling laws shows quantitative relationships between model size, data volume, compute, and prediction error; from this it follows that larger models and more data typically generalize better, up to unexpected new abilities at very large scale (Kaplan et al., 2020; Wei et al., 2022). It is important to emphasize that such observations are empirical and do not guarantee unlimited improvements; costs and ecological footprint also increase with size.

Part 3 (approx. 10 minutes): Applications, Limits and Thought Exercises

Concrete applications where pretraining is central: text generation (autocomplete, creative text), dialog systems (chatbots), machine translation, question answering and assistant systems, code generation and information extraction. In many cases the pretrained model is subsequently fine-tuned or adapted via instruction- or feedback-based methods to make it more useful for a specific task (Brown et al., 2020; Ouyang et al., 2022).

Typical limits and risks in practical use: (1) Models can produce false or unsupported statements with high confidence (hallucinations). (2) They reflect societal biases and other distortions from the training data. (3) They are sensitive to the form of the input (prompt design) and to shifts between training and deployment data. (4) Privacy risks and direct reproduction of sensitive content can occur if such content was present in the training data. These problem areas are widely discussed in research (Bender et al., 2021; Lin et al., 2022).

Three short thought exercises for deepening:

First exercise: Take a short, open sentence beginning (e.g., "The temperature rose because…") and consider which continuations are particularly likely. Discuss what kinds of knowledge the model needs to choose the most likely continuation, and which kinds of knowledge would not follow from statistical regularity alone.

Second exercise: Formulate an example where two different but commonly occurring text sources could report conflicting facts (e.g., different years). Consider how a pretrained model would generate an answer and what postprocessing measures or additional data would be needed to obtain more reliable information.

Third exercise: Briefly discuss which measures (e.g., data cleaning, targeted fine-tuning, human feedback) are realistic to vet a model before deployment and reduce possible harm. Which limitations nevertheless remain?

In conclusion: The scientific consensus views next-token prediction as a surprisingly effective but inherently statistical learning signal. It produces powerful representations, but not automatically reliable, explainable, or morally unproblematic behavior; practice requires additional measures, evaluation and transparency (Brown et al., 2020; Bender et al., 2021; Ouyang et al., 2022).