Lecture 27: Memory

Part 1 (approx. 20 minutes) — Basic understanding: Why do agents need memory?

A "memory" in an AI agent is the ability to retain information beyond single time points and retrieve it later. This can be illustrated by the distinction in human behavior: short-term memory (what I just heard), working memory (what I'm currently using to compute), and long-term memory (facts, routines, experiences). For AI agents, memory serves similar functions: it enables coherence over longer interaction sequences, supports planning, and improves adaptation to environments that are only partially observable.

Concrete reasons why agents need memory:

1) Partial observability: Many real situations are POMDPs (partially observable Markov decision processes). An agent does not see the entire world in a single step; stored observations or formed beliefs are needed to improve current decisions. This principle is well established in the reinforcement learning literature.

2) Long-range dependencies and planning: Some goals require remembering earlier events, e.g., a robot that must execute a sequence of steps, or an assistant that must maintain consistency over a long conversation. Without memory, only very short context dependencies are possible.

3) Personalization and continuity: Services that adapt user preferences over weeks or months need persistent storage of preferences, consents, and allowed actions.

4) Efficiency through caching and preprocessing: Caching results (e.g., retrieval of relevant documents, vector-based caches) saves compute time and repetition. This is a practical form of memory used in retrieval-based architectures.

Analogy: An agent without memory is like a conversational partner who forgets what was said after every sentence — this leads to fragmented interactions. An agent with appropriate memory mechanisms can selectively use relevant past information, like a well-organized notebook.

Open points: There is no single "optimal" memory for all tasks. Different applications require different memory dimensions (timescale, granularity, access speed, privacy). These trade-offs are central to current research.

Part 2 (approx. 20 minutes) — Terms, concepts and mechanisms

The following introduces central terms and describes typical technical variants of memory mechanisms.

Key terms

Episodic memory: Storage of individual experiences or observations with timestamps. Episodes can be retrieved later to respond to similar situations (e.g., experience replay in reinforcement learning).

Semantic memory: Abstract, generalized facts or models about the world (e.g., "coffee mugs are fragile"). Such information supports generalization and transfer.

Working/short-term memory: Short-lived information needed for ongoing tasks (e.g., the most recently mentioned context in a conversation).

External vs. internal memory: Internal memory is embedded in a model's parameters (weights) or active states; external memory is maintained as a separate data structure and queried as needed (examples: vector databases, key-value stores, external attention mechanisms).

Techniques and architectural principles

1) Experience replay: In many RL algorithms, past transitions are stored and reused for training to reduce correlations and make sampling more efficient. This principle is established in RL teaching and also helps stabilize learning processes.

2) Retrieval-supported systems: For knowledge-intensive tasks, external documents or vector entries are retrieved by relevance and presented to the model as context. Such hybrids combine a parameterized model with a non-parametric knowledge base and allow updates without full retraining.

3) Memory-Augmented Neural Networks: There are architectures that explicitly use an addressable memory (key/value structures) to read and write information. This idea has been formalized in several influential works and shows how models can learn to use external memory instead of only changing weights.

4) Consolidation and compression: Raw data can often be transformed through abstraction and compression into more informative, persistent representations (e.g., deriving semantic rules from individual episodes). Such processes reduce memory requirements and can improve robustness.

What should be stored? Criteria for selection

Relevance: Information that is likely to improve future decisions is a candidate for storage. Relevance can be estimated automatically via usage or predictive importance.

Temporal durability: Some information is only useful short-term (conversation context), other information is long-term (user preferences). A memory policy must distinguish between these categories.

Privacy and legality: Only store information whose storage is legally permissible and for which the user has consented.

Storage cost vs. benefit: A simple economic trade-off — storage and queries have costs (storage space, latency, privacy risks). Design must weigh these costs against expected benefits.

When is forgetting useful — mechanisms of forgetting

Forgetting is not only information loss but often an active mechanism to improve performance:

- Staleness (becoming outdated): Old information can be misleading if the environment has changed. Automatic expiration dates or age-based weighting reduce wrong decisions.

- Capacity limits: With limited storage space, schemes like least-recently-used (LRU), importance-based removal, or stratified admission rules determine which entries are kept.

- Protection against overfitting and catastrophic forgetting: In continual learning, strategies such as selective rehearsal, regularization, or separate memories for old tasks are needed to avoid overwriting important knowledge.

Practical procedures: temporal weighting, sample replay, explicit delete commands, aggregation of old data into condensed form.

Uncertainty: The optimal balance between retaining and forgetting is an active research question; there is no universally best mechanism because domain requirements vary widely.

Brief references to underlying technical works

The principles described above for using external memory, as well as retrieval hybrids, are documented in the ML literature. Key concepts like retrieval-augmented generation and addressable external memory structures form the technical basis of most current memory solutions, especially for knowledge-intensive applications.

Part 3 (approx. 10 minutes) — Applications, limits and thought exercises

Practical examples:

1) Conversational assistant with long-term memory: An assistant stores user preferences (e.g., preferred language, dislikes) and uses them to produce consistent and personalized responses. Here, storing user data is linked to clear consent and transparent deletion mechanisms.

2) Household robot: Episodic memory enables remembering locations of objects, successful tactics and hazard spots. The robot benefits from compressed representations (e.g., maps) rather than raw sensor data.

3) Knowledge-intensive question answering: Retrieval-based systems (e.g., hybrid models) use external document collections that can be updated regularly to provide fact-based and up-to-date answers.

Limits and risks:

- Faulty memory contents can promote misbehavior (e.g., persistent false facts). Verifying stored information is therefore relevant.

- Privacy and compliance: Persistently stored personal data require technical and organizational measures (consent, deletability, minimization).

- Scalability: Large external memory volumes require efficient indexing and retrieval strategies; simple linear search does not scale.

Summary: Memory is central for agents to achieve coherence, planning, personalization and efficiency. Which information is retained and for how long is a design- and domain-specific trade-off. Forgetting is as important a tool as storing. Many practical solutions combine parameterized models with external, updatable memories.