Part 1 (20 minutes): Data as the most important resource — Explanation with analogies and examples
Duration: approx. 20 minutes
In discussions about artificial intelligence, people often first talk about models, architectures and computing power. From a practical perspective and in many recent publications, however, it is undisputed that data play the central role: models learn from examples, and the quality, quantity and composition of these examples largely determine how well a system later performs in practice. A common analogy compares data to soil for plants: even the best seed and the most sophisticated irrigation are of little use if the soil is depleted or toxic. A related analogy is that of fuel: models are engines that only deliver performance if the fuel — the training data — has an appropriate composition and purity.
Concrete examples make this relationship visible. Language models today are trained on extremely large text collections; open corpora such as "The Pile" document how diverse such collections can be and that they contain substantial amounts of unstructured web data, books and other sources (Gao et al., 2020). For multimodal models, image‑text pairs are used in very large quantities; examples are the LAION datasets, which have shown how freely available, web‑based image‑text pairs can provide a basis for powerful image‑text models (Schuhmann et al., 2021). For large, proprietary models like GPT‑4, the publishers describe that a mixture of publicly available, licensed and human‑curated data is used; at the same time, the exact composition of proprietary training corpora often remains not fully public (OpenAI, 2023).
From a practical point of view, a paradigm shift has emerged in recent years in favor of what is called "data‑centric AI": instead of primarily developing new architectural ideas, developers and researchers increasingly focus on systematically improving training data — through better labeling, cleaning and consistency of the data (deeplearning.ai course materials). This shift is not a dogmatic exclusion of model improvements, but an indication that many sources of errors in ML systems can be traced back to the data (Sculley et al., 2015).
Part 2 (20 minutes): Deepening — Technical terms and properties of good training data
Duration: approx. 20 minutes
For a precise discussion it is helpful to clarify central technical terms. Training data are the examples a model sees during its learning process. Validation and test data are independent subsets used for selecting hyperparameters and for final assessment of generalization, respectively. Label quality denotes the correctness and consistency of annotated target values. Dataset‑shift describes the shift in distribution between training and deployment data; models often generalize worse when such shifts occur (Goodfellow et al., 2016).
What properties should a dataset have to be considered "good"? First, accuracy and reliability of labels are central: systematic mislabels lead to persistently incorrect conclusions. At the same time, experimental work shows that deep learning models are robust to a certain extent against random label noise, but targeted or systematic errors can have more severe effects (Rolnick et al., 2017). Second, representativeness is important: the training data should adequately reflect the diversity of real‑world use cases. Lack of representation leads to unfair or erroneous outcomes, as exemplified by studies of face recognition algorithms that performed significantly worse for certain groups (Buolamwini & Gebru, 2018).
Third, cleanliness and consistency of the data are relevant: distant duplicates, incorrect metadata or mixed encodings create noise and can make training inefficient and error‑prone. Fourth, documentation and provenance are crucial: knowledge about origin, licensing and possible limitations of the data is necessary to manage legal and ethical risks; works debating large, web‑based corpora emphasize both technical performance gains and ethical questions around provenance and representation (Bender et al., 2021).
Other useful terms concern the data‑quantity versus quality debate. Large amounts of raw data have undeniably contributed to the performance improvement of modern language models (see C4 in the T5 project and The Pile), yet increasing the dataset alone is not a panacea: data must be relevant and not only plentiful. Technical reports of large models document mixtures of politically neutral texts, licensed collections and curated examples, and they also point to filtering and cleaning processes that were applied (Raffel et al., 2019; OpenAI, 2023).
Practical consequences of poor data appear on several levels. For model performance, poor data quality leads to lower accuracies, unstable generalization and increased susceptibility to failures under new conditions. In production systems, additional "technical debt" arises: constant rework of data pipelines, debugging of misclassifications and maintaining manual review processes, as described by Sculley et al. (Sculley et al., 2015). On a societal level, biased training corpora can lead to discriminatory outcomes, privacy violations and misinformation; this is relevant both technically and ethically (Buolamwini & Gebru, 2018; Bender et al., 2021).
Finally, the question of sufficient availability of high‑quality data is not conclusively answered. There are large open collections for text and images (The Pile, C4, LAION), and they have significantly enabled research and development (Gao et al., 2020; Raffel et al., 2019; Schuhmann et al., 2021). Nevertheless, many specially required domain datasets — for example medical imaging with verified diagnoses or trustworthy legal corpora — are not unrestrictedly available because they are rare, regulated or expensive. In addition, there is uncertainty about the exact composition of commercial training corpora, which is why statements about general data availability should be made with caution (OpenAI, 2023). Indications of concrete attack surfaces, such as models vulnerable to membership‑inference attacks, further show that not only quantity but also confidentiality of the data matters (Shokri et al., 2017).
Part 3 (10 minutes): Application, limits and thought exercises
Duration: approx. 10 minutes
To consolidate what has been learned, we consider typical applications and the associated limits. In automated industrial image classification, the same problems often occur: insufficiently representative training images lead to failures with unusual camera angles or certain material properties. In face recognition, empirical work has shown that unbalanced training data lead to significantly worse results for women and dark‑skinned people; the case makes clear that technical failure and societal impact are connected (Buolamwini & Gebru, 2018). In language models, problems appear as hallucinations or factually incorrect answers; technical reports on large models document how different data sources, cleaning and filtering procedures are tried to reduce these risks, even though not all problems are fully solved (OpenAI, 2023; Bender et al., 2021).
A practical approach to improvement is data‑centric: instead of only tweaking architecture and optimizer, it is often worthwhile to check data labels, standardize rules for data collection, remove duplicates and analyze systematic biases. Such steps are part of recommended practice in course materials and in industry projects aiming at reliable production (deeplearning.ai; Sculley et al., 2015).
Finally, three short thought exercises for self‑assessment: First: You are to build a classification model for medical images. What priorities do you set: as many automatically collected, poorly labeled images as possible, or fewer but carefully expert‑verified cases? Second: You deploy a language model in a legal advisory scenario. What requirements for provenance, timeliness and liability of the data would you define? Third: An existing system produces erroneous predictions for a new user group. How would you determine whether this is dataset‑shift or a model problem, and which data‑related measures would you prioritize?
These tasks are not intended to have definitive answers; they are meant to show that data questions should be placed earlier in system development. Engineers and decision‑makers should be aware of the limits of data: even excellent models can fail in application without appropriate, documented and verified data. At the same time, there is research and development need in methods for efficient data improvement, better dataset documentation and procedures that jointly consider quality, fairness and privacy.