Lecture 18: Fine‑Tuning

Part 1 (approx. 20 minutes) — Core idea, analogies and examples

Foundation models and large language models are trained in an elaborate pretraining phase on general text tasks, typically with the objective of predicting the next token or solving similar general language tasks. This pretraining gives the model a broad, statistical understanding of language, syntax and many facts, similar to a general education or schooling. Nevertheless, this general education is not sufficient when concrete, specialized tasks are expected, such as medical diagnoses in text form, legal contract analysis, or company-specific customer support flows.

A helpful analogy is the training and further education of a person: pretraining corresponds to a general university education in which broad foundations are taught; fine‑tuning is comparable to professional training or specialist medical training, in which specific skills, terminology and procedures are practiced. Without this specialization, a person with broad knowledge may not reliably meet the requirements of a demanding specialized field.

In practice, this is evident because pretraining makes a model "useful" for many possible texts and tasks, but not necessarily optimal for a single, clearly defined task. Examples from research illustrate this: BERT gained strong general language representations through pretraining, but the best results for tasks such as question answering or text classification were achieved only after additional task-specific training (fine‑tuning) on labeled data (Devlin et al., 2018). Similarly, work on large autoregressive models shows that pretraining provides good few‑shot capabilities, but in many practical application scenarios additional adaptation is needed to achieve desired behaviors stably and reproducibly (Brown et al., 2020).

Important reasons why pretraining alone is often insufficient are therefore: (1) objective mismatch between the pretraining task and the concrete target task, (2) domain shift between the training corpus and the target application, and (3) need for behavior adaptation, for example regarding safety and formatting requirements. These points are repeatedly cited in the literature as central motivations for fine‑tuning and other adaptation methods (Bommasani et al., 2021).

Key literature references: Vaswani et al. (2017); Devlin et al. (2018); Brown et al. (2020); Bommasani et al. (2021).

Part 2 (approx. 20 minutes) — Methods and terminology clearly introduced

Terminologically, several variants of specializing a pretrained model are distinguished in practice. The most important are: full fine‑tuning, domain-adaptive pretraining, parameter-efficient methods (Adapter, LoRA, Prompt‑Tuning), instruction tuning and RLHF (Reinforcement Learning from Human Feedback). I explain each method, how it works and typical pros and cons.

Full fine‑tuning means that nearly all weights of the pretrained model are further trained on a labeled dataset. This is conceptually simple and often yields very good results for a specific task, but requires a lot of compute and storage, because a complete copy of the model weights can be created for each new task (Goodfellow et al., 2016).

Domain-adaptive pretraining (also called "continued pretraining") describes an intermediate step: the model is further pretrained before the final fine‑tuning using large, unlabeled texts from the target domain to align the model's language distribution with that domain. This can improve the effectiveness of subsequent task-specific adaptations, especially when the domain differs significantly from the general training data (Bommasani et al., 2021).

Parameter-efficient methods do not change all weights but add compact module structures or adaptation vectors. Examples are Adapters, where small additional layers are inserted between existing layers and only these are trained (Houlsby et al., 2019); LoRA (Low‑Rank Adaptation), which constrains weight updates via low-rank factorizations so that few parameters are changed (Hu et al., 2021); and Prompt‑Tuning, where fixed or learnable input prompts are prepended instead of modifying the model internals (Lester et al., 2021). These approaches save memory and allow managing many different specializations without duplicating the entire model for each specialization.

Instruction‑Tuning refers to fine‑tuning on datasets that train behavior under instructions, i.e., contain examples such as an input task and the desired, often natural-language form of the output. Studies and development reports show that instruction‑tuning makes models more robust at following instructions (Ouyang et al., 2022). A related step is RLHF, in which human preferences are used to optimize model behavior via reinforcement learning techniques; this addresses soft preferences and safety-relevant aspects that are difficult to capture in simple supervised setups (Ouyang et al., 2022).

Important technical terms and challenges: "distributional shift" (domain shift between training data and deployment data), "overfitting" (adapting to idiosyncrasies of the training dataset, reducing generalization), "catastrophic forgetting" (forgetting previously learned capabilities during further training), "parameter efficiency" (proportion of parameters changed) and "calibration" (how well model uncertainties align with actual error probability). Catastrophic forgetting has been described in the literature as a real phenomenon and there are methods that address this problem with regularization or special replay mechanisms (Kirkpatrick et al., 2017). Overfitting and regularization are fundamental concepts in machine learning education (Goodfellow et al., 2016).

Practical criteria play a role in methodological decisions: available labeled data, compute budget, need for data isolation for privacy (e.g., data must not alter the base model), maintainability of multiple model variants and requirements for interpretability or safety guarantees. Vendor documentation summarizes practical guidelines on data formats, validation and cost/benefit of fine‑tuning (OpenAI Fine‑Tuning Guide).

Key literature references: Houlsby et al. (2019); Hu et al. (2021); Lester et al. (2021); Ouyang et al. (2022); Kirkpatrick et al. (2017); Goodfellow et al. (2016); Bommasani et al. (2021); OpenAI Fine‑Tuning Guide.

Part 3 (approx. 10 minutes) — Applications, limits and thought exercises

Applications that particularly benefit from fine‑tuning are found where specific, reproducible formatting or domain requirements exist. Examples include text classification with concrete label sets (e.g., sentiment, topic classification), named‑entity recognition and other sequence-labeling tasks in a domain context, question‑answering systems with high precision requirements, text summarization for specific text types (medical reports, legal texts) and generative systems that must adhere to formal output formats (e.g., structured responses, JSON schemas). For many of these tasks, fine‑tuning increases accuracy and consistency compared to pure prompting or pure pretraining (Devlin et al., 2018; Brown et al., 2020; Ouyang et al., 2022).

Limits and disadvantages of fine‑tuning are also important: it can lead to overfitting to the training style, reducing robustness to slightly altered inputs. Fine‑tuning can also cause catastrophic forgetting of earlier capabilities if appropriate measures are not taken; that is, a model may sometimes lose more general knowledge while adapting to a specific task (Kirkpatrick et al., 2017). Furthermore, privacy and safety risks arise if training data contain sensitive information and the model reproduces this information in responses or, under certain prompts, emits "memorized" training fragments; research shows that large models can reproduce parts of their training corpus under certain conditions (Bommasani et al., 2021). Fine‑tuning can also require substantial compute and is organizationally demanding when many specialized variants must be managed.

Small thought exercises to consolidate learning:

1) Imagine you are to adapt a pretrained model for a medical question‑answering system, and you have few labeled examples but access to many unlabeled medical texts. Which steps would you prioritize and why? (Hint: domain-adaptive pretraining can align the distribution; Adapter or LoRA methods are parameter-efficient, reduce storage needs for multiple specializations and can help limit privacy risks.)

2) Suppose a fine‑tuned model begins to reproduce confidential data from the training corpus in its answers. What measures are possible to reduce this risk? (Hint: data sanitization, differential privacy during training, using parameter-efficient modules instead of full fine‑tunes, and evaluative tests for memorization risks are discussed in the literature.)

3) Consider when prompt engineering and instruction‑tuning on the one hand and full fine‑tuning on the other are preferable. Which operational conditions (e.g., resources, number of specializations, requirements for verifiability) influence this decision?

In conclusion, the scientific consensus is that fine‑tuning remains a central method to make pretrained language models usable for specific practical requirements; at the same time, the choice of method and the management of associated risks are decisive and context-specific (Bommasani et al., 2021; Ouyang et al., 2022).

Key literature references: Devlin et al. (2018); Brown et al. (2020); Ouyang et al. (2022); Bommasani et al. (2021).