Chapter 3: Neural Networks: Perceptrons and Activations

From sparse vectors to learned representations

TF-IDF and bag-of-words turn documents into sparse vectors. Each dimension is a single word, so two different words live in completely different places. The model has no way to learn that "car" and "automobile" are similar, or that "king" and "queen" differ in a consistent, meaningful direction. It can only count exact matches.

To capture meaning, we need dense, continuous representations that are learned from data. Neural networks let us do exactly that. They convert discrete tokens into vectors of numbers, combine those vectors through learned weights, and produce a probability distribution over the next token. The same network can learn that related words should have similar vectors.

This chapter builds a neural network from the smallest unit, the perceptron, up to a simple multi-layer classifier. We will see how weights, biases, activation functions, and layers fit together, and why a stack of simple units can represent surprisingly complex functions.

Why neural networks?

Imagine a vocabulary of 50,000 words. A bag-of-words vector has 50,000 dimensions, and the vector for "cat" and "kitten" are orthogonal: every dimension is different. A neural network instead learns a smaller vector for each word, perhaps 128 or 512 dimensions, by adjusting the numbers so that words with similar contexts end up near each other.

The same idea applies beyond words. A network can learn that a document about machine learning should have a similar internal representation to a document about deep learning, even if the two documents share few exact words. It learns the representation from the data rather than engineering it by hand.

The perceptron

A perceptron takes a vector of inputs, multiplies each by a weight, sums them, adds a bias, and passes the result through a step function. It is a binary classifier: is the point above or below the line?

Here x is the input vector, w is the weight vector, b is the bias, and sigma is an activation function. For a simple perceptron, sigma is a step function; for modern networks, it is a smooth non-linearity such as ReLU or sigmoid.

The weights determine which inputs matter and in which direction. A positive weight means "more of this input pushes the output up"; a negative weight means the opposite. The bias lets the decision boundary move away from the origin. Without it, every separating line would have to pass through zero.

import numpy as np

# A simple perceptron for AND-like logic
x = np.array([1.0, 0.5, -0.3])
w = np.array([0.4, 0.6, -0.2])
b = -0.1

z = np.dot(w, x) + b
y = 1 if z > 0 else 0
print("weighted sum:", z, "output:", y)

Run the code and change the inputs. The output is one if the weighted sum is greater than zero and zero otherwise. That is the perceptron decision rule.

From one neuron to many

A single perceptron can only learn linearly separable functions. It cannot solve XOR, where the two classes are intertwined. The fix is to add more neurons and arrange them in layers.

A hidden layer of neurons transforms the input into a new representation. With enough hidden units, a multi-layer network can approximate any continuous function. The first layer might learn edges, the second might combine edges into shapes, and later layers might combine those shapes into concepts. In language models, early layers learn morphology and syntax, while deeper layers learn semantics and reasoning.

Each layer applies a linear transformation followed by a non-linear activation. The output of one layer becomes the input of the next. Stacking layers lets the network build hierarchical features. This universal approximation property is why deep networks are so powerful: enough neurons and layers can represent almost any function we care about.

The XOR problem

XOR is the classic example of a non-linearly separable problem. The inputs (0, 0) and (1, 1) should give output 0, while (0, 1) and (1, 0) should give output 1. No single straight line separates the two classes.

A multi-layer network can solve XOR by first learning two hidden features, then combining them. For example, one hidden neuron can detect "at least one input is one" and another can detect "both inputs are one". The output neuron then activates when the first is true but the second is false. This is only possible because the hidden layer re-represents the input.

Activation functions

Activation functions introduce non-linearity. Without them, stacking many layers would be no more expressive than a single layer because a composition of linear functions is still linear. The choice of activation shapes how easily the network trains and what kinds of functions it can learn.

• Sigmoid: maps any value to (0, 1). Historically popular but suffers from vanishing gradients for very positive or very negative inputs.
• ReLU: max(0, x). Fast to compute and avoids vanishing gradients for positive inputs, which is why it dominates modern deep learning.
• Tanh: maps to (-1, 1), zero-centered, which can make optimization easier than sigmoid.
• Softmax: turns a vector of logits into a probability distribution over classes.

import numpy as np

def relu(x):
    return np.maximum(0, x)

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

x = np.array([-2.0, -0.5, 0.0, 0.5, 2.0])
print("relu:", relu(x))
print("sigmoid:", sigmoid(x).round(3))

Softmax is the final layer of most language models. It converts raw scores for each vocabulary word into probabilities that sum to one. The choice of activation inside the hidden layers affects how easily the network trains and how well it generalizes.