Lecture 31: Security — Limits of Artificial Intelligence

Part 1 (approx. 20 minutes) — Introductory presentation: What does "security" mean for AI?

When we speak of security for AI systems, we mean several interconnected aspects: the reliability of outputs (robustness), protection against targeted manipulation (attacks on models or data), the protection of sensitive data (privacy), and the accountability of those who develop and operate systems. A helpful everyday analogy is that of a car: a car must be mechanically robust, must not be easily manipulated from the outside, should protect the privacy of its occupants, and the manufacturer bears responsibility for design, quality assurance and instructions for use. Accordingly, AI also needs "vehicle inspection" (testing procedures), "locks and sensors" (security mechanisms) and rules for manufacturers and operators.

Practical examples make the problems tangible. Adversarial examples are artificially altered inputs that can cause a model to reach incorrect conclusions, even though humans interpret them correctly. Data poisoning refers to targeted manipulation of training data to influence later model behavior. With "model extraction" attackers can reconstruct a model from an accessible interface (API), thereby undermining trade secrets or security features. For personal data there is the risk that private information can be reconstructed from a model's use. These forms of attack have been demonstrated and documented in research; there is scientific consensus that they pose real risks for which technical countermeasures are necessary.

At the same time it is important to emphasize where uncertainties lie: some attacks only work under certain assumptions (e.g., full knowledge of the model or access to training data), and the effectiveness of countermeasures can depend on architecture, data quality and the deployment environment. Current guidelines such as the NIST AI Risk Management Framework summarize the state of research in a practical way and recommend risk analyses and process design as core tasks to mitigate dangers.

Part 2 (approx. 20 minutes) — Deepening: Technical terms, attack types and protection mechanisms

Important attack types

Adversarial inputs (adversarial examples) describe targeted, often small-seeming modifications to input data that mislead a model while appearing unchanged to humans. Research has shown several techniques to construct such examples, as well as countermeasures like adversarial training, which prepares models for such manipulations. While adversarial training improves robustness in tested scenarios, full immunity against all possible manipulations has not yet been achieved.

Data poisoning concerns manipulations of the training dataset. Such manipulations can be subtle (backdoors/trojans: the model learns an ingrained trigger behavior) or broad, aiming to degrade performance. Poisoning is especially relevant when training data come from insecure sources or when public/user-generated data are used without strict quality control.

Model theft (model extraction) often exploits a publicly accessible query interface to reconstruct a functionally similar model. This can allow proprietary models to be copied, security barriers to be bypassed, or enable additional attacks such as adversarial examples to be prepared more easily.

Attacks against privacy

Membership inference attacks attempt to determine whether a specific record or individual was included in a model's training set. Model inversion attacks aim to infer training data from a model and thereby reconstruct sensitive information. Both attack types have been practically demonstrated in the literature and show that models without protections can violate privacy.

Defense strategies and their limits

Technical countermeasures exist in several categories. Differential Privacy is a mathematically defined concept that limits the probability with which information about individual training examples can be reconstructed from model outputs; practical implementations for deep learning are available, but they involve trade-offs between privacy and model performance. Adversarial training increases robustness against certain types of manipulations; its effectiveness depends on the spectrum of attacks simulated during training. Other measures include input sanitization, monitoring of requests to models (to detect extraction attempts), and the use of access controls and logging.

In addition to technical means, research recommends process measures: systematic risk analysis, documented testing, red-teaming and continuous monitoring in production. Institutional frameworks such as the NIST AI Risk Management Framework recommend treating security not as a one-time task but as an iterative, organization-wide process.

Role of governance and standards

Alongside technical solutions, regulatory and normative requirements form the third pillar of security. International principles, such as the AI principles formulated by the OECD, and legislative initiatives like the European Commission's proposal for the "Artificial Intelligence Act", aim to set minimum requirements, transparency obligations and responsibilities. Such regulatory frameworks concern, for example, classification of high-risk systems, reporting obligations for security incidents and requirements for data quality and documentation. The concrete design and impact of these regulations are subjects of ongoing political and professional debate; as a fact it can be stated that such frameworks are currently being developed and implemented in many jurisdictions.

Responsibility of developers and operators

Developers and organizations bear responsibility in several dimensions: they must identify risks, implement adequate countermeasures, test and document their models, and be transparent about limits and possible malfunctions. This responsibility also includes organizational measures such as security-by-design, data governance and clear processes for updates and incident response. Guides and frameworks from research and standards bodies recommend assigning responsibilities clearly and conducting security reviews throughout the development lifecycle.

Transparency about uncertainties

Many defensive measures do not provide absolute protection but rather reduce probabilities or limit damage under certain assumptions. Research is active: new attack scenarios and defense approaches are continuously published. There is therefore scientific agreement that AI security requires ongoing review, adaptation and external evaluation.

Part 3 (approx. 10 minutes) — Applications, limits and thought exercises

To connect the previous points, we consider three brief application examples and the associated security issues. First: a medical assistance system that supports diagnoses from X-ray images. Here the risks of adversarial manipulation (false-positive/false-negative misdiagnoses) and privacy breaches are acute. Countermeasures include robust testing, access restrictions, differential privacy for training data and clear documentation of system limitations.

Second: a chatbot that processes customer data and is available via an API. Relevant dangers are model extraction, prompt injection (inputs that cause the system to behave undesirably) and leakage of confidential information. Practical protections are rate limiting, output filtering, logging, more tightly controlled contexts for sensitive queries and targeted red-teaming.

Third: a quality inspection system in a production line. Here robustness against unusual sensor values or targeted disturbances is central. Strategies include robust sensor data fusion, anomaly detection and regular tests under realistic disturbance conditions.

Small thought exercises for deeper engagement: 1) Imagine you operate a public classification API. What indicators in operation would signal that a model-extraction attack is taking place? 2) You are to train a model with sensitive patient data: what trade-offs would you make between model quality and the privacy parameters of a differential privacy solution? 3) Your team discovers that a model systematically reacts incorrectly to certain input variants: what steps would you take to identify, fix and communicate this externally?

For each of these tasks the same holds: technical measures, organizational processes and transparent communication are required. The combination reduces risks but does not eliminate them completely; therefore responsibilities, monitoring and contingency plans must be part of the system.