Chapter 5: Neural Networks: Backpropagation

The credit assignment problem

A deep network may have millions of weights, and the loss at the output depends on all of them. When the prediction is wrong, which weights are responsible? Backpropagation answers this question by applying the chain rule from calculus, working backward from the loss to every parameter in the network.

Without backpropagation, training deep networks would be computationally impossible. We would have to recompute the loss many times, varying one weight at a time. Backpropagation computes all the gradients in one forward pass and one backward pass.

The chain rule

A neural network is a composition of functions. Layer 1 feeds into layer 2, layer 2 feeds into layer 3, and so on. The chain rule says that the derivative of a composition is the product of the derivatives of each step. We multiply small, local derivatives along the path from the loss back to each weight.

In a network, this multiplication happens layer by layer. The loss signal flows backward, picking up a local derivative at every operation. That is why the algorithm is called backpropagation.

Computational graphs

It helps to think of a network as a computational graph. Each node is an operation: a matrix multiplication, an addition, an activation, or a loss. Each edge is a tensor that flows forward. During the backward pass, we reverse every edge and propagate gradients.

A node receives a gradient from downstream, computes its own local derivatives, and sends gradients to its inputs. Matrix multiplication nodes transpose the appropriate weight matrix. Activation nodes multiply by the derivative of the activation. Loss nodes produce the first gradient.

Forward and backward passes

Forward pass: data flows through each layer to produce a prediction and a loss.

Backward pass: the loss signal returns through the layers in reverse. Each layer computes gradients for its own weights and biases, then passes the signal on to the previous layer.

At each layer, multiply the incoming signal by the activation derivative to get the local error. Weight gradients are the local error times the layer's input. This pattern repeats all the way back to the input.

A worked example

Consider a tiny network with one input x, one weight w, one bias b, and a sigmoid activation. The prediction and the mean-squared-error loss are shown below. We want the gradients dL/dw and dL/db.

Forward pass:

Backward pass through each function:

This three-step chain is exactly what backpropagation automates. In a deep network, the chain is longer, but each step is the same: multiply by the local derivative and continue backward.

Backpropagation in code

The code below walks through a 2-layer network: two inputs, three ReLU hidden units, and one sigmoid output. The forward pass produces a prediction and loss. The backward pass produces gradients for every weight.

import numpy as np

# A tiny 2-layer network: 2 inputs -> 3 hidden (ReLU) -> 1 output (sigmoid)
x = np.array([[0.5], [0.2]])           # shape (2, 1)
y = 1.0                                 # target

W1 = np.array([[1.0, 0.0],
               [0.0, -1.0],
               [0.0, 0.5]])             # shape (3, 2)
b1 = np.array([[-0.1], [0.0], [0.0]])  # shape (3, 1)

W2 = np.array([[1.0, -2.0, 3.0]])      # shape (1, 3)
b2 = np.array([[-0.7]])                # shape (1, 1)

# ---- Forward pass ----
z1 = W1 @ x + b1
a1 = np.maximum(0, z1)                  # ReLU
z2 = W2 @ a1 + b2
a2 = 1.0 / (1.0 + np.exp(-z2))          # sigmoid
loss = -y * np.log(a2) - (1 - y) * np.log(1 - a2)

# ---- Backward pass ----
# 1. Output error signal
delta2 = a2 - y                         # for sigmoid + cross-entropy

# 2. Gradients for the last layer
dW2 = delta2 @ a1.T
db2 = delta2

# 3. Propagate the error signal to the hidden layer
relu_prime = (z1 > 0).astype(float)
delta1 = (W2.T @ delta2) * relu_prime

# 4. Gradients for the first layer
dW1 = delta1 @ x.T
db1 = delta1

print("a2:", a2.item())
print("loss:", loss.item())
print("dW2:", dW2.flatten())
print("dW1:", dW1)

Study the shapes: x is (2, 1), W1 is (3, 2), so delta1 is (3, 1). Multiplying delta1 by x.T gives dW1 of shape (3, 2), matching W1. This shape bookkeeping is the heart of backpropagation.

Vanishing gradients

When a network is deep, gradients can become very small by the time they reach the early layers. Sigmoid and tanh have derivatives close to zero for large positive or negative inputs, so the chain rule multiplies many small numbers together. This is the vanishing gradient problem.

ReLU helps because its derivative is one for positive inputs, so gradients do not shrink. Residual connections and layer normalization, which we will see in Transformers, also make it easier to train deep networks by giving gradients alternate paths to flow.

Initialization matters

If all weights start at zero, every neuron computes the same thing and learns nothing. If weights are too large, the first forward pass produces huge activations and gradients. If they are too small, signals die out through many layers.

Good initialization balances the variance of activations across layers. He initialization scales weights by the square root of the fan-in, keeping the signal strength roughly constant. Proper initialization is a small detail that makes a large difference in whether a deep network trains at all.