Part 1 (20 minutes): Basic understanding with analogies and examples
Duration: 20 minutes
When a neural network "learns", this generally means: it changes its internal parameters — the weights and biases — so that its outputs for specific tasks better match the desired answers. Crucial to this is that the network first produces an output via a forward pass and then this output is compared to the desired target; the discrepancy is measured with a so-called loss function. This difference is the error visible to humans. The central question that backpropagation answers is: How do you distribute this error across the many individual parameters of the network so that you can adjust them purposefully?
A simple analogy is adjusting a group of screws that together control a machine. One observes the final result; if it is wrong, one must determine which screws to change and by how much to improve the result. Backpropagation provides a systematic method to estimate for each screw (each weight) how strongly a small change of that weight affects the overall result. Technically this is done via derivatives: one computes how the loss function changes if a single weight is changed infinitesimally. These derivatives are called gradients.
Concrete example: A simple network with one hidden layer takes input values and computes the output using weighted sums and activation functions. After comparing with the target, an error results. Backpropagation computes, in a backward pass, how this error flows back step by step to earlier layers, where in each layer the local sensitivity (the derivative of the activation function) is multiplied by the error that has already flowed back. This is a direct application of the chain rule from differential calculus. The original idea of propagating errors layer by layer is a core contribution of early work on error backpropagation in neural networks and was established in the 1980s (see Rumelhart, Hinton & Williams).
Once the gradients are computed, the weights are adjusted: the usual scheme is "subtract the gradient multiplied by a learning rate" — formally: new weight = old weight − (learning rate × gradient). In practice this is done in small steps, often on mini-batches of examples, to balance computational cost and noise effects. The smaller steps also explain why learning proceeds incrementally: large steps often "overshoot" and worsen performance; small steps allow a controlled approach to better solutions.
The described basic idea and the algorithmic implementation are presented in detail and instructively in freely available textbooks and course materials (see in particular Goodfellow, Bengio & Courville as well as the course materials from CS231n and Michael A. Nielsen).
Part 2 (20 minutes): Deeper dive — technical terms and mathematical foundations
Duration: 20 minutes
To formalize backpropagation, we introduce some terms and explain their role in the algorithm. The loss function L(z, y) measures the discrepancy between the model prediction z and the desired target y. The model prediction z is the result of the forward pass, where from inputs x the output is computed by successive linear operations (weighted sums) and nonlinear activations. A weight w is a parameter that we want to adjust.
The goal of backpropagation is to compute the partial derivatives ∂L/∂w for all parameters w efficiently. Direct computation would be too expensive with many parameters; backpropagation uses the chain rule to compute derivatives recursively from the output back to the input. Formally this means: if a layer computes the activation a = f(u) with u = W·x + b, then for the derivative of the loss with respect to W the product rule yields ∂L/∂W = (∂L/∂a) · (∂a/∂u) · (∂u/∂W). The quantities ∂L/∂a are taken from the next layer that has already been considered; this creates an efficient backward recursion.
The numerical optimization step by which weights are actually adjusted is often a form of the gradient method. The simplest variant is gradient descent with learning rate η: w ← w − η ∂L/∂w. In practice the stochastic variant (Stochastic Gradient Descent, SGD) has become common, where the gradients are estimated from subsets of the training data (mini-batches). Advanced optimizers like Momentum, RMSprop or Adam build on the same gradient information but weight or normalize it adaptively; textbooks and course materials provide detailed comparisons of these extensions (see Goodfellow et al.; CS231n).
Important terms commonly used here include: activation function (e.g., sigmoid, tanh, ReLU), gradients, learning rate, mini-batch, epoch (pass through the full training set), overfitting and regularization. The choice of activation function directly influences the derivatives; e.g., S-shaped activations (sigmoid, tanh) can produce derivatives close to zero, which leads to attenuation of the error when propagating backward. This phenomenon is called vanishing gradient; its opposite, exploding gradient, describes diverging gradients. Both problems are well documented and constitute an important practical aspect when training deep networks (see Goodfellow et al.; LeCun et al.).
Why do incremental updates work from a mathematical perspective? Gradients give the direction of steepest ascent of the loss function; the negative direction is the direction of local improvement. Since the loss landscape of a deep network is not convex, each individual gradient vector only provides local information — it is a linear approximation of how the loss changes in the neighborhood of the current parameter. Small steps are based on the assumption that this local linear approximation remains valid for small changes; larger steps can violate this approximation and thus lead to worse results.
Practical tips to stabilize and accelerate training include normalization methods (e.g., Batch Normalization), activation functions with favorable derivative properties (e.g., ReLU) and architectural elements like residual blocks that help preserve gradients across many layers (see He et al., 2015). These measures reduce in practice the effects of vanishing/exploding gradient problems, but they do not fully explain why deep networks often generalize well — this remains an active research area (see Goodfellow et al.).
Part 3 (10 minutes): Applications, limitations and thought exercises
Duration: 10 minutes
Backpropagation is the basis for training modern neural models in a wide range of applications: image and speech processing, machine translation, medical image analysis, recommender systems and many other fields. In all cases, backpropagation enables end-to-end adjustment of model parameters for specific tasks, often with large datasets and high computational resources.
At the same time, backpropagation has limitations. First, success depends strongly on the quality and quantity of data; poor or biased training data lead to corresponding errors in the model. Second, there are algorithmic problems such as vanishing or exploding gradients that can make training very deep networks difficult; various technical countermeasures have been developed for these (ReLU, Batch Normalization, residual connections). Third, there is no complete theoretical understanding of why and under which conditions deep networks generalize so well; the theoretical analysis of optimization landscapes and generalization is the subject of ongoing research (see Goodfellow et al. for a summary of open questions).
Finally, some short thought exercises to deepen understanding: First, consider a very small network with only one weight w and a quadratic loss function. In which direction should w be adjusted if the gradient is positive? Second, think of a multi-layer chain of functions; how does the chain rule explain that a derivative near zero in an early activation function hinders the training of the weights before it? Third, consider the consequences of a learning rate that is too large or too small for the training process, and which practical methods one can use to select the learning rate.
These questions can be approached with the principles described above; detailed introductions and exercise examples are contained in the referenced teaching materials and books.