Lecture 19: RLHF — Why human feedback for language models?

Part 1 (20 minutes) — Core idea, motivation and illustrative examples

Large language models are trained by pretraining on large amounts of text so that they can predict the next word. This objective alone makes the models good pattern generators, but not automatically good conversational partners or reliable helpers for specific human goals. Models trained solely by next-token prediction can produce answers that are factually incorrect, dangerous, rude, or simply unsuitable for a particular use case.

The core idea of RLHF (Reinforcement Learning from Human Feedback) is to use humans to guide a model toward which of the possible answers are desirable. A helpful analogy is training an assistant: First the assistant learns general language (pretraining). Then you give example instructions and correct answers (supervised fine-tuning). Finally, humans compare alternative answers and indicate which are better; from this a scoring measure (reward) is learned, and the model is adjusted to produce answers that receive higher scores according to this measure. This multi-stage approach is common in practice and has been described in technical reports and papers (cf. Ouyang et al., 2022; Christiano et al., 2017).

Concrete example: For the task "Summarize this text briefly" a pretrained model may produce long, unclear, or irrelevant summaries. Humans compare different summaries pairwise and choose the better one. From these pairwise decisions a reward model is trained that provides a better approximation of human preferences. Afterwards the language model is adjusted using RL methods so that it produces answers with higher reward values. Publications report that perceived quality and usefulness of answers improve in this way (cf. Stiennon et al., 2020; Ouyang et al., 2022).

Why is human feedback necessary? In short: Because humans have complex, often not formally defined preferences (e.g., appropriateness, brevity, politeness, accuracy) that are difficult to encode into a loss function for pretraining. Human feedback provides direct signals about these preferences that a pure next-token objective does not provide.

Part 2 (20 minutes) — Technical details and terminology

The RLHF pipeline can be divided into three main steps, as described in technical papers (Christiano et al., 2017; Ouyang et al., 2022):

1) Supervised Fine-Tuning (SFT): Based on human demonstrations or corrected answers, the model is first brought closer to the desired behavior via supervised learning. This step stabilizes the model and provides a good initial policy for later optimization.

2) Reward Modeling: Humans compare model answers pairwise or rate answers on a scale. From these comparison data a separate model is trained that predicts a numerical "reward" for a given input and output. This reward model is an estimate of human preferences and replaces direct but expensive human evaluation during the RL process.

3) Reinforcement Learning (Policy Optimization): The language model (policy) is updated using an RL method to produce actions (token sequences) that receive high scores according to the reward model. In practice Proximal Policy Optimization (PPO) or a related policy-gradient method is often used. It is common to include regularization, for example a Kullback-Leibler (KL) penalty relative to the starting policy, so that the model does not drift too far from the SFT checkpoint and thereby lose language quality (cf. Ouyang et al., 2022).

Important terms (short definitions):

Policy: The model that, given an input, produces a distribution over possible outputs. In the context of language models the policy is the probability distribution over the next tokens.

Reward Model: A model that predicts a numerical score approximating human preferences. The reward model is trained from human pairwise comparison data or ratings (Christiano et al., 2017).

Preference Data: Pairwise comparison or rating data produced by humans who compare two or more answers and select the better one. Such data form the basis for training the reward model.

Policy Optimization / PPO: RL algorithms that change the policy to maximize expected reward. PPO is commonly used in practice because it allows more stable updates than simple policy-gradient methods.

KL Regularization: An additional constraint that ensures the updated policy does not deviate too far from the original policy. This regularization reduces quality degradation and prevents excessive exploitation of the approximated reward model (Ouyang et al., 2022).

Key advantages reported in the literature: RLHF can steer models toward desired behaviors that are hard to formalize; it improves usability in dialogue and instruction-following tasks and reduces certain undesired responses, provided the reward model and preference data are representative (Ouyang et al., 2022; Stiennon et al., 2020).

At the same time there are technical pitfalls and limitations discussed in research: Reward hacking (the model finds a strategy that yields high estimated reward but does not meet human intent), reliability of the reward model (it can learn systematically wrong preferences), scalability (human evaluations are expensive), bias in preferences (human annotators are not neutral) and distributional shift (the policy generates responses outside the distribution the reward model was trained on). Many of these points are explicitly addressed in foundational works (Christiano et al., 2017; Ouyang et al., 2022).

For context: Reinforcement learning concepts such as reward, policy and value are formalized in textbooks (Sutton & Barto, 2018). RLHF uses these concepts but typically replaces the classical, explicitly defined reward function with a learned, human-based reward model.

Part 3 (10 minutes) — Applications, limitations and discussion prompts

Concrete applications: A prominent example is the InstructGPT/related instruct versions of large language models, which were optimized via RLHF to better follow instructions and provide more helpful answers (see Ouyang et al., 2022; OpenAI Blog on the Instruct series). In research RLHF has also been successfully used to align automatic text summarization with human preferences (Stiennon et al., 2020) and in numerous other instruction-following and dialogue scenarios.

Major limitations and open problems:

- Measurability and generalization: The reward model is an approximation; if the policy pursues new, unexpected strategies, the reward model may rate them incorrectly (distributional shift). This point is emphasized in both methodological papers (Christiano et al., 2017) and applications (Ouyang et al., 2022).

- Cost and scalability: High-quality human preference data are expensive to collect. The literature discusses various compromises, such as active selection of examples to be rated or the use of annotator models, but a complete solution is lacking.

- Bias and divergence of human preferences: Preferences are culturally and contextually dependent; a reward model reflects the biases of the designers and raters. This is a documented risk in the application of RLHF.

- Safety and "reward hacking": A model can find ways to maximize the approximated reward without fulfilling the underlying human objective. Such failure modes are both practically documented and theoretically anticipated (Christiano et al., 2017; discussions in Ouyang et al., 2022).

Short discussion prompts (for in-class discussion or short assignments):

1) You are to design a reward model for polite email responses. Which questions would you ask raters to generate pairwise comparison data, and what risks do you see? (Expected considerations: precision of instructions to raters, definition of politeness, representativeness and bias.)

2) Name a possible form of "reward hacking" for a summarization task and propose a mitigation measure. (Example consideration: the model produces very short, content-poor summaries that the reward model rates as concise; mitigation: explicit quality checks in reward data collection or additional terms in the reward model.)

3) What are the pros and cons of replacing human ratings with automated annotator models? (Expect: cost reduction vs. possible amplification of errors and biases.)

In conclusion: RLHF is a practical, currently widespread tool to align language models more closely with human preferences. It alleviates many typical problems of purely data-driven models but creates new challenges regarding robustness, measurability and ethical implications. The findings and limitations mentioned are based on the experiences and experiments documented in the cited literature (Christiano et al., 2017; Stiennon et al., 2020; Ouyang et al., 2022).