Lecture 29: Energy Demand of Modern AI Models — Limits of Scaling

Part 1 (approx. 20 minutes): Basic understanding — Why do modern AI models consume so much energy?

Modern AI models, especially large deep learning models, require large amounts of computational power. This is not primarily a consequence of "inefficient code", but follows from three factual factors that interact: 1) the scale of the mathematical operations, 2) the frequency of these operations during training and inference, and 3) the surrounding infrastructure (data centers, cooling, networks). Long‑term developments show a strong increase in required compute over recent years; trend analyses document an exponential rise in compute used in leading projects (see OpenAI, "AI and Compute") [OpenAI 2018].

An illustrative analogy: training a large model is comparable to building and testing a complicated engine in a variety of variants. Each training run is a test run, and the more parameters ("adjustment screws") a model has, the more test runs are required so the engine runs reliably. Unlike engine construction, in AI the number of parameters, the amount of training data and the number of runs (epochs) are very large, which is why energy demand increases.

It is important to distinguish between training and inference: training (adjusting the model weights) is very compute‑intensive and often a one‑time or rare activity during a model's lifetime. Inference (applying the model to inputs), however, can occur over long periods and in large numbers, and thus cumulatively cause significant energy demand. Whether training or inference accounts for the larger share of energy in the long run depends on the use case: for a model queried millions of times per day, inference can dominate; in research scenarios with many training runs, training can be dominant (overview and discussion in Strubell et al. 2019 and Patterson et al. 2021) [Strubell et al. 2019; Patterson et al. 2021].

Finally, one must emphasize the role of infrastructure: compute does not run in isolation. Data centers require energy for power delivery as well as for cooling and other operational tasks. The efficiency of a data center is commonly measured by the metric PUE (Power Usage Effectiveness); a lower PUE means less additional overhead relative to pure compute power (see The Green Grid; IEA) [The Green Grid; IEA 2021].

Part 2 (approx. 20 minutes): Technical depth — Where does the greatest energy consumption occur and which technical terms are central?

The direct source of energy consumption in AI is the computational operations. Two concepts are a starting point here: model size (number of parameters) and compute cost (measured for example in FLOPs, i.e., floating‑point operations). Larger models with more parameters typically produce more FLOPs per forward or backward pass; additionally, with larger models the number of training examples and epochs often increases. OpenAI documents the strong increase in compute used in leading examples over the years ("AI and Compute") and thus makes the systematic rise in compute demand visible [OpenAI 2018].

Hardware layer: modern training setups use specialized accelerators like GPUs or TPUs. This hardware offers high compute per watt, but it also has a significant manufacturing and replacement component ("embodied emissions"), whose detailed quantification is still incomplete. Measurements of the efficiency of individual accelerators and the performance of data center deployments are documented in the technical literature (e.g., Jouppi et al. on TPUs) [Jouppi et al. 2017].

Infrastructure layer: the total energy demand of a model consists of the energy for the compute operations plus data center overhead. PUE is a common metric here. In addition, local CO2 emissions depend strongly on the regional electricity mix: diesel, coal, gas or a high share of renewables change the climate impact of the same amount of electricity considerably. Patterson et al. discuss the importance of location and electricity source for the total emissions of a training job and propose reporting standards to improve comparability [Patterson et al. 2021].

Energy‑intensive phases: in training the repeatedly executed backward and forward passes of networks are particularly dominant. In modern transformers, matrix multiplications within the attention and feed‑forward blocks dominate the FLOPs. With large datasets and many training steps, these operations add up to very large compute times. In deployment (inference) costs rise with the number of queries; efficient architecture and system decisions can yield large savings here (e.g., batch processing, caching).

Measures at different levels to increase efficiency: Algorithmic: model compression (distillation), quantization (floating‑point to integer formats), pruning (removing unimportant weights) and more efficient architectures such as sparser attention mechanisms. Examples and evaluations can be found among others in work on DistilBERT and in surveys on efficient transformers (Sanh et al. 2019; Tay et al. 2020) [Sanh et al. 2019; Tay et al. 2020].

Systemic / infrastructure: better hardware, intelligent job scheduling (e.g., shifting training jobs to times with lower grid CO2 intensities), data center location choice and improved cooling reduce the CO2 emitted per kWh. A practical example is the use of machine learning to optimize cooling in existing data centers, with documented energy savings (DeepMind/Google) [DeepMind 2016].

Reporting and scientific practice: several contributions call for standardized measurements and transparency: information on energy consumption, hardware type, runtime, location and energy sources is necessary to make claims about environmental impacts robust. Strubell et al. and Patterson et al. emphasize this need and propose reporting standards (the "Green AI" approach; Green AI argues for measuring and publishing energy metrics alongside pure performance metrics) [Strubell et al. 2019; Patterson et al. 2021; Schwartz et al. 2019].

Uncertainties and data gaps: so far there is no comprehensive, standardized catalog that provides complete lifecycle accounting for all large‑scale AI projects (including hardware manufacturing, operation, disposal). Many publications report only partial data (e.g., GPU hours or rough CO2 estimates). Therefore, comparability and aggregation in the scientific community are still limited (Patterson et al. 2021 explicitly point to these limitations) [Patterson et al. 2021].

Part 3 (approx. 10 minutes): Applications, limits and small thought exercises

Concrete applications illustrate the different burdens: a research project that runs many training jobs with different hyperparameters has high one‑time training consumption. A production chat service with millions of queries per day, by contrast, causes ongoing inference costs that can be large cumulatively. Recommendation and personalization systems can be particularly energy‑intensive because they require very frequent inference and possibly frequent model updates.

Ecological impacts can be divided into two categories: direct emissions from electricity consumption (dependent on electricity generation) and indirect/structural effects such as increased hardware demand and associated manufacturing and disposal requirements. The IEA has shown that data centers and data networks already account for a noticeable share of global electricity consumption and that efficiency improvements as well as shifts in usage are important to cushion the growth in load (IEA 2021) [IEA 2021].

Limits of current practice: even with more efficient algorithms there remain trade‑offs between model performance and energy cost. In addition, many existing ecosystems are optimized for short‑term performance metrics, not for lifecycle emissions. The call for more transparency, standardized reporting and the inclusion of energy efficiency as a metric in research is described as necessary by several authors (Strubell et al., Schwartz et al., Patterson et al.) [Strubell et al. 2019; Schwartz et al. 2019; Patterson et al. 2021].

Three short thought exercises to consolidate learning: 1) Assume you operate a service with 1,000 requests per day. Consider which measures at the model, system and infrastructure levels could quickly lead to noticeable energy savings and why. (Hint: caching, batch inference, smaller models and quantization are typical levers.)

2) Consider which information you would need to publish at minimum to make the CO2 footprint of a training run comparable. (At minimum: compute time on specified hardware, location/grid CO2 intensity during runtime, PUE of the data center.)

3) Discuss in which cases it may be more ecologically sensible to train a large model multiple times (e.g., for many individual customizations) instead of using a single general model that is trained once expensively and then adapted via fine‑tuning. (The outcome depends heavily on the frequency of uses, the extent of the customizations and the possibility of reuse.)

Final summary: the energy demand of modern AI models is a multi‑layered problem that requires algorithmic, systemic and political/organizational solutions. Technical advances in hardware and more efficient algorithms can achieve a lot, but transparency and systematic reporting are needed to enable robust comparisons and thus informed decisions. The scientific literature shows consensus that efficiency must become a central metric and that concrete measures at all levels (algorithm, hardware, data center, energy mix) are necessary to limit environmental impact (see Strubell et al., Schwartz et al., Patterson et al., IEA) [Strubell et al. 2019; Schwartz et al. 2019; Patterson et al. 2021; IEA 2021].