Chapter 2: Neural Networks from Scratch

From counts to neurons

N-gram models memorize exact strings. To overcome this, we want a model that learns continuous representations of words and contexts. Neural networks let us do exactly that. They convert discrete tokens into vectors of numbers, combine those vectors through learned weights, and produce a probability distribution over the next token.

This chapter builds a neural network from the smallest unit, the perceptron, up to a simple multi-layer classifier. We will see how weights, biases, activation functions, loss, and gradients fit together.

The perceptron

A perceptron takes a vector of inputs, multiplies each by a weight, sums them, adds a bias, and passes the result through a step function. It is a binary classifier: is the point above or below the line?

Here x is the input vector, w is the weight vector, b is the bias, and sigma is an activation function. For a simple perceptron, sigma is a step function; for modern networks, it is a smooth non-linearity such as ReLU or sigmoid.

Activation functions

Activation functions introduce non-linearity. Without them, stacking many layers would be no more expressive than a single layer because a composition of linear functions is still linear.

• Sigmoid: maps any value to (0, 1). Historically popular but suffers from vanishing gradients.
• ReLU: max(0, x). Fast and avoids vanishing gradients for positive inputs.
• Tanh: maps to (-1, 1), zero-centered.
• Softmax: turns a vector of logits into a probability distribution.

Softmax is the final layer of most language models. It converts raw scores (logits) for each vocabulary word into probabilities that sum to one.

Loss functions and cross-entropy

Training needs a measure of how wrong the model is. For classification, we use cross-entropy loss. It compares the model's predicted probability distribution to the one-hot target distribution.

When the target is one-hot, only the log-probability of the correct class matters. If the model is confident and correct, the loss is near zero. If it is confident and wrong, the loss grows large.

import numpy as np

def softmax(z):
    e = np.exp(z - np.max(z))
    return e / e.sum()

def cross_entropy(y_hat, y):
    # y is one-hot, y_hat is softmax probabilities
    return -np.sum(y * np.log(y_hat + 1e-9))

logits = np.array([2.0, 1.0, 0.1])
probs = softmax(logits)
print(probs)

Gradient descent

To minimize the loss, we compute the gradient of the loss with respect to each weight. The gradient points in the direction that increases the loss, so we move the weights a small step in the opposite direction. Repeating this on many examples is gradient descent.

In practice, we rarely use the full dataset at once. Stochastic gradient descent uses one example at a time, and mini-batch SGD uses a small batch. Batches give a noisy but useful estimate of the true gradient and allow hardware parallelism.

Backpropagation

Backpropagation is the chain rule applied to a computational graph. It computes gradients efficiently by reusing intermediate values. First, a forward pass computes the loss. Then, a backward pass applies the chain rule from the loss back to every parameter.

# a tiny computational graph: y = (x + b) * w
x, b, w = 2.0, 1.0, 3.0

# forward
t = x + b
y = t * w

# backward from y
dy_dt = w
dt_db = 1.0
dt_dx = 1.0

dy_db = dy_dt * dt_db  # 3
dy_dx = dy_dt * dt_dx  # 3