Lecture 26: Retrieval-Augmented Generation (RAG)

Part 1 (approx. 20 minutes) — Why trained knowledge is often insufficient and a conceptual introduction to RAG

Language models are trained on large amounts of text. This training produces a parameterized model that encodes patterns, formulations, and many facts in its weights. However, this form of internal knowledge has systematic limits: the training corpus is temporally limited, it does not cover all special cases, and scaling it to reliably "encode" a very large number of rare facts is costly. Furthermore, facts change over time; a model based only on a static snapshot of data will lag behind current events. These problems are identified in the research literature as central motivations for retrieval-supported methods (cf. Lewis et al., 2020; Guu et al., 2020).

A useful analogy is that of a library plus a librarian: a purely parametric model corresponds to an expert who has read a lot and remembered much, but does not carry every single book from the library with them. Retrieval-Augmented Generation (RAG) combines this internal expert with fast access to external books (a document collection). Instead of answering solely from "memory," the system fetches relevant sources from an index and uses those texts as support when generating a response. This makes it possible to make more current, specific, or comprehensive knowledge available without retraining the entire model (Lewis et al., 2020; Guu et al., 2020).

Practical example: For a question about current guidelines from a medical professional society, a pure language model would provide formulations based on its training state that may be outdated. A RAG system, by contrast, can search a daily-updated index for the relevant guideline fragments and explicitly incorporate that information into generation, which can improve the factuality of the output (Lewis et al., 2020).

It is important to note that retrieval does not automatically guarantee perfect answers. Quality depends on the completeness and quality of the index, the representations used for search (e.g., keyword-based or dense vectors), and the generator's ability to correctly integrate retrieved texts. These dependencies are core topics of the following sections (Karpukhin et al., 2020; Hugging Face documentation).

Part 2 (approx. 20 minutes) — Technical operation and terminology

Retrieval-Augmented Generation is an architectural principle with two clear functional parts: the retriever and the generator (also called the reader). The retriever searches an external knowledge base and returns a small set of relevant documents or text fragments. The generator takes these hits as additional context and produces the answer based on them. The basic idea has been elaborated in several research works; important contributions include REALM (Google Research, 2020), Dense Passage Retrieval (DPR, Karpukhin et al., 2020) and the RAG framework (Lewis et al., 2020).

About the retriever: There are classical keyword-based methods like BM25 as well as newer dense methods that embed texts and queries as vectors in a high-dimensional space. Dense methods use embedding models that can capture semantic similarity better than simple keyword overlap; a prominent example of this approach is DPR, which trains question and passage encoders separately to find matching passage vectors (Karpukhin et al., 2020). For searching large corpora, these vectors are stored in vector indexes and efficiently searched with libraries like FAISS (Facebook AI Research, FAISS-GitHub).

About the generator: Modern RAG variants couple the retriever with an autoregressive language generator (e.g., Transformer-based models). Two variants described in the original paper are RAG-Sequence and RAG-Token. In RAG-Sequence, the generator produces a probability distribution over answers for each read passage and then combines these; in RAG-Token, contributions from different passages are combined already during the generation of individual tokens. Both approaches allow generation to be directly influenced by the retrieved texts; the RAG paper explains pros and cons and shows training options, such as end-to-end optimization when retriever and generator components are made jointly differentiable (Lewis et al., 2020).

Indexing and infrastructure: A practical RAG system requires a maintained document index, mechanisms for updating, and an efficient vector search service. FAISS is a common open-source library for approximate nearest neighbor search that serves as a basis in many implementations. The quality of the index crucially determines the relevance of retrieved evidence; incomplete or poorly segmented texts lead to irrelevance and thus to erroneous answers (Facebook AI Research; Hugging Face documentation).

Evaluation and reliability: In benchmark tests, RAG has been able to improve accuracy on knowledge-intensive tasks compared to purely parametric models, because concrete text excerpts are available as evidence for answers. However, benchmarks are only one aspect; in real applications, robustness, latency, and the ability to correctly cite sources play a major role. Research points out that retrieval increases factuality but also introduces new error sources — for example when irrelevant or contradictory passages are retrieved (Lewis et al., 2020; Karpukhin et al., 2020).

Part 3 (approx. 10 minutes) — Applications, limitations and small thought exercises

Applications: RAG approaches are used particularly in open-domain question answering, in assistive research systems, in customer-specific chatbots with corporate knowledge, and in summarization tasks when external texts are consulted for answering. The ability to index an internal company document set or a daily-updated knowledge collection makes RAG attractive for production solutions because the generated output can more easily be traced back to concrete sources (Hugging Face documentation; Lewis et al., 2020).

Key limitations and risks: First, retrieval remains error-prone; a poor index or unsuitable embeddings lead to irrelevant hits. Second, RAG users face latency and cost: vector search and repeated context processing increase computational demand. Third, RAG does not fully protect against hallucinations: if the generator freely combines information or draws conclusions not supported by the retrieved documents, false outputs remain possible — RAG reduces this risk but does not eliminate it (Lewis et al., 2020). Fourth, there is organizational overhead: index maintenance, updates, access control and data protection must be operationally solved, especially for sensitive corporate data. Finally, there are legal and ethical questions about source usage, citability and responsibility in case of errors.

Small thought exercises for consolidation:

1) Imagine a RAG-based customer service center uses product documentation from one year ago as its index. Briefly describe three concrete error sources that can arise from outdated documents, and propose two measures to mitigate them. (Hint: Think about static facts, price changes and new safety warnings.)

2) A RAG system returns an answer with citations from multiple documents that contradict each other. What strategies could you provide the generator so that it recognizes or explicitly flags contradictions? (Hint: Explicitly referencing sources, uncertainty phrasing, restraint in making definitive statements.)

3) To what extent does a dense vector index (DPR) help compared to a pure keyword search (BM25) for semantically formulated queries? Discuss advantages and disadvantages depending on domain and data quality.

These thought exercises are intentionally open; there are no blanket solutions, but they are meant to encourage connecting the technical mechanisms with organizational requirements and critically examining the remaining uncertainties of RAG-based systems.