Part 1 (20 minutes): Intuition, analogies and simple examples
Gradient descent is a fundamental method by which many learning algorithms in artificial intelligence adjust a model's parameters step by step so that an error measure (the loss function) becomes smaller. In simple terms: a model "learns" by making small changes to its settings that reduce the error. The changes are made iteratively, not all at once. (See introductions to optimization and deep learning literature, in particular Goodfellow et al. 2016; CS231n notes.)
A common analogy is that of a hiker on a foggy mountain: the hiker wants to descend into the valley (the valley represents low error), but can only see the immediate surroundings. They feel their way downhill step by step, choosing at each step the direction that goes downhill the steepest. In mathematics this direction corresponds to the negative gradient of the function: the gradient shows the direction of steepest ascent, and its negative shows the direction of steepest descent (see presentations of the gradient concept in optimization textbooks).
Another picture: place a ball in a curved bowl. The ball rolls down until it comes to rest in a depression. For simple, convex bowls that depression is the global minimum; for irregular, mountainous landscapes the ball can become stuck in a local hollow. This behavior explains why iterative descent methods do not always lead to the best possible solution (see Goodfellow et al., Boyd & Vandenberghe).
A concrete, easy-to-follow example: with a linear regression model and mean squared error, the error can be expressed as a function of the model parameters. Gradient-descent updates change the weights until a minimum is reached. For quadratic, convex error surfaces the method reliably converges to the global minimum; for more complex models like deep neural networks the error surface is no longer convex and the method has different properties (see CS231n; Goodfellow et al.).
Important initial terms that already appear here: loss function (loss), parameters/weights (weights), update step, learning rate (step size). A practical distinction to note is between full batch gradient descent (all training examples per update) and stochastic or mini-batch based gradient descent (subsets of examples per update); the latter are common in large-data scenarios (see Bottou et al. 2018; CS231n).
Source note for this section: Goodfellow, Bengio & Courville (2016); CS231n notes (Stanford); Bottou, Curtis & Nocedal (2018).
Part 2 (20 minutes): Technical terms and deeper examination
What exactly is a gradient?
Formally, the gradient of a differentiable function L(θ) with respect to a vector parameter θ is the vector of all partial derivatives: ∇L(θ) = (∂L/∂θ1, ∂L/∂θ2, ...). It gives the direction of steepest ascent in parameter space. Gradient descent uses the negative gradient as the direction vector for the next update. In simple form the update rule is:
θ ← θ − η ∇L(θ),
where η is the learning rate (step size). The magnitude of η determines how far a single step in parameter space goes.
Learning rate and step-size issues
Choosing the learning rate is central: if η is too large, the algorithm can overshoot the minimum and diverge; if η is very small, convergence will be very slow. Modern methods use adaptive schemes that adjust the step or take past updates into account, such as momentum, RMSProp or Adam. These methods are discussed in the literature and tested empirically (see Kingma & Ba, 2015; Goodfellow et al., 2016).
Convex vs. non-convex optimization
For convex functions there are mathematically grounded statements about convergence of gradient methods; under suitable conditions one converges to the global minimum (see Boyd & Vandenberghe, 2004). For non-convex functions, as occur in deep neural networks, these general guarantees do not exist: one can at best show that iterations under certain conditions converge to stationary points (points where the gradient is zero), but not that these stationary points are global minima (Goodfellow et al.; Bottou et al.).
Typical difficulties in learning
The literature identifies several recurring problems:
- Local minima and saddle points: Non-convex landscapes often contain many stationary points; saddle points are particularly numerous in high-dimensional landscapes and can slow progress (Goodfellow et al.; Bottou et al.).
- Plateaus and flat regions: Areas with very small gradient lead to very slow adaptation (Goodfellow et al.).
- Vanishing/exploding gradients: In deep networks derivatives can become extremely small or large, especially with certain activation functions or weight initializations; this impedes effective learning (Goodfellow et al.; Nielsen).
- Noisy gradients with stochastic updates: Stochastic gradients have higher variance; this can make the trajectory noisy but can also help escape local minima (Bottou et al.; CS231n).
- Hyperparameter dependence: Performance is sensitive to learning rate, batch size, regularization, etc.; this often requires experimental tuning.
Improvements and extensions
Based on the mentioned difficulties, numerous methods have been developed: momentum mechanisms store a running average of previous updates to stabilize the direction; adaptive methods adjust learning rates per parameter (Adam is a widespread example; Kingma & Ba, 2015). These methods have practical advantages in many problems, but they also bring their own questions (e.g. effects on generalization) that are being investigated in research (Goodfellow et al.; Bottou et al.).
Source note for this section: Boyd & Vandenberghe (2004); Goodfellow, Bengio & Courville (2016); Bottou, Curtis & Nocedal (2018); Kingma & Ba (2015); CS231n notes; Nielsen (online).
Part 3 (10 minutes): Applications, limits and short thought exercises
Applications
Variants of gradient descent are the backbone of training many practical models: from simple logistic regression to support vector machines (in certain solution methods) to deep neural networks for image and speech processing. In large applications, typically a form of stochastic or mini-batch gradient descent with additional techniques (momentum, adaptive learning rates, regularization) is used to make computation and memory efficient and to obtain robust results (CS231n; Bottou et al.; Goodfellow et al.).
Major limitations
Important, evidenced limitations are: first, there is no general guarantee to find the global optimum for non-convex problems; second, success strongly depends on data quality, model choice and hyperparameter settings; third, numerical and algorithmic issues (e.g. very small gradients) can prevent learning. These limitations are the subject of active research and caution is advised in application (Goodfellow et al.; Bottou et al.; Boyd & Vandenberghe).
Short thought exercises / exercises
- Consider a simple one-dimensional function L(θ) = θ^2. How does gradient descent behave here? What role does the learning rate η play? (Hint: ∇L = 2θ.)
- Why can stochastic gradient descent help to escape a local minimum that becomes a trap for batch gradient descent? Formulate the intuition in terms of noise in the updates.
- You observe that training error stagnates for a long time while validation error rises. What plausible causes exist, and what short- to mid-term measures could you take?
Short solutions / hints: For 1) The updates are θ ← θ − 2ηθ, which leads to multiplicative damping; for |1−2η|<1, θ converges to 0. For 2) The noise can move the trajectory over small energy barriers; for 3) One possible reason is overfitting — measures: increase regularization, adjust the learning rate, get more data or use early stopping. These hints are based on standard presentations in textbooks and review articles (Goodfellow et al.; Bottou et al.; CS231n).
Source note for this section: CS231n; Goodfellow, Bengio & Courville; Bottou, Curtis & Nocedal; Kingma & Ba.