Lecture 7: Neural Networks

Introduction and Context

Neural networks are a class of machine learning models that consist of many interconnected computing units. The models are historically inspired by the structure of biological nervous systems, but they use heavily simplified components and mathematical learning methods. The fundamental idea is that through the interaction of many simple processing units complex input–output relationships can be learned. Important modern textbooks and reviews systematically summarize fundamentals and principles (cf. Goodfellow, Bengio & Courville 2016; Michael Nielsen 2015) [1][2].

Part 1 (20 minutes): Basic understanding with analogies and examples

Goal: Provide an understandable basic concept.

An appropriate analogy is that of a jury made up of many members: Each member (an artificial neuron) looks at the evidence (inputs), weights different arguments differently (weights) and gives a verdict (output). Individual jurors are simple, but the collective decision can be very nuanced. In a neural network, inputs in the form of numbers (for example pixel values of an image) are passed to many neurons. Each neuron computes a weighted sum of its inputs, adds a bias term and applies a nonlinear function that decides how strongly it "fires" or what output it produces. This nonlinear transformation is crucial: without nonlinearity, a cascade of layers would be only a linear mapping and could not represent complex patterns (cf. Nielsen 2015; Goodfellow et al. 2016) [2][1].

As a concrete example, one can imagine image recognition. In early layers neurons learn simple orientation patterns or edges, further inside combinations of these features form textures or corners, and in very deep layers there are more abstract features such as parts of an object. The clearly visible structure of this hierarchy has been repeatedly observed in practice and is one reason why multilayer networks (deep networks) are so powerful (cf. Stanford CS231n) [3].

It should be noted that the biological analogy should not be taken literally. Biological neurons, synapses and learning mechanisms are considerably more complex than the mathematical models. The inspiration is functional: principles such as local processing, connections with changeable strength and hierarchical processing served as a model, not as a direct mapping of physiological details (cf. Britannica on Hebb's rule and introductions) [8][9].

Part 2 (20 minutes): Technical terms and how it works

Goal: Introduce important terms cleanly and explain the learning mechanism.

An artificial neuron is often referred to as a perceptron when it uses a simple threshold function; historical introductions can be found under the term "Perceptron" (cf. Britannica) [9]. In general usage an artificial neuron denotes a unit that computes a weighted sum of its inputs x, adds a bias b and applies an activation function f, so that the output is y = f(w·x + b). Typical activation functions are sigmoid functions, hyperbolic tangent and today very often the so-called ReLU (rectified linear unit), because it offers practical advantages when training large networks (cf. Goodfellow et al. 2016) [1].

Networks are distinguished by their architecture: feedforward networks pass information only forward, while recurrent networks have feedback and are suitable for temporally ordered data. Special layers like convolutional layers use local connection patterns and shared weights, making them particularly efficient for image data; this is described formally and practically in course materials on Convolutional Neural Networks (cf. Stanford CS231n) [3].

The actual learning is done by adjusting the weights so that a predefined objective is achieved. One defines a loss function that measures how far the current outputs are from the desired outputs. Using gradient methods one computes the derivative of the loss function with respect to the weights and shifts the weights in the direction of the negative gradient. The efficient computation of these derivatives in networks with many layers is done by the backpropagation algorithm, whose operation and mathematical basis are presented in detail (cf. Goodfellow et al. 2016; Nielsen 2015) [1][2].

Why many layers? Formally there are statements that networks with more layers can represent certain functions more compactly (with fewer parameters) than very wide but shallow networks. Practically, multiple layers enable the stepwise formation of abstract representations: each layer extracts features from the representation of the previous layer. Theoretical and empirical works argue that depth creates favorable structures for learning complex functions, even if a shallow model could mathematically represent the same functions, but often with greater effort (cf. Goodfellow et al. 2016) [1].

Important terms you should place here are: weights, bias, activation function, layer, forward pass, loss function, gradient descent, backpropagation, regularization (measures against overfitting) and special layer types such as convolutional or attention mechanisms. Modern architectures extend these building blocks, for example through self-attention, as described in the publication "Attention is All You Need", which has become relevant for many language processing tasks (cf. Vaswani et al. 2017) [6].

Part 3 (10 minutes): Applications, limits and thought exercises

Goal: Show concrete applications, name limitations and provide short exercises for consolidation.

Neural networks have proven particularly suitable for a number of tasks. Convolutional Neural Networks have been successfully applied in image processing and object recognition; a historically significant milestone was the use of deep convolutional networks for the ImageNet competition, which demonstrated that deep networks can achieve much better results for large image classification tasks (cf. Krizhevsky, Sutskever & Hinton 2012) [4]. For sequential data and game control, combinations of deep networks and optimization methods have led to the discovery of powerful strategies; a well-known example is the combination of deep learning and reinforcement learning to control agents that could reach human-level performance on Atari games (cf. Mnih et al. 2015) [5]. In language processing, transformer models with attention mechanisms have made great advances (cf. Vaswani et al. 2017) [6].

At the same time there are clear limitations. Neural networks often require large amounts of labeled data or elaborate training procedures, and they are prone to overfitting to training data. Another practical problem are so-called adversarial examples: small, targeted changes to inputs can mislead models, which points to structural weaknesses and is an active field of research (cf. Goodfellow, Shlens & Szegedy 2014) [7]. Moreover, many current architectures are statistical pattern followers: a deeper understanding of causality, robust common-sense knowledge or explainable reasoning is not automatically provided. The science of biological learning mechanisms is also incomplete; the simplicity of artificial neurons stands in contrast to the complexity of biological synapses and networks, which is why direct inferences to the brain should be viewed with caution (cf. Britannica on Hebb and on Perceptron; Goodfellow et al. 2016) [8][9][1].

For consolidation I suggest three short thought exercises: (1) Consider which preprocessing steps are sensible when designing a network for classifying handwritten digits (e.g. scaling, normalization, data augmentation). (2) Discuss the advantages and disadvantages of local connections and shared weights in convolutional layers compared to fully connected layers. (3) Assess the implications of adversarial examples for safety-critical applications and which measures (e.g. robustness tests, ensemble methods, regularization) can be taken. For all three points there is extensive practical and theoretical literature; an introduction can be found in the mentioned textbooks and course materials [1][2][3][7].

Conclusion

In summary, neural networks provide a flexible, powerful class of models that can solve complex tasks through simple computing units and learnable connections. The biological inspiration provides useful concepts, yet the mathematical models are highly abstracted. Deep architectures enable hierarchical representations, which explains many practical successes, while challenges remain regarding data requirements, robustness and explainable generalization. The cited sources provide both the intuitive and the formal foundations and point to current research questions (see especially Goodfellow et al. 2016; Nielsen 2015; Stanford CS231n) [1][2][3].