Learn LLM

What is a language model?

A language model is, at its core, a probability distribution over sequences of words or tokens. Given a sequence like "the cat sat on the", a language model assigns a probability to every possible next token: "mat", "floor", "moon", or even "banana". The ones that sound more plausible get higher probability.

This simple idea is the engine behind auto-complete, machine translation, speech recognition, and modern chatbots. If we can estimate P(next token | previous tokens) well, we can generate coherent text by repeatedly sampling the most likely next token.

From probabilities to text

A language model does not understand meaning the way humans do. It captures statistical patterns: which words tend to follow which contexts. Surprisingly, when this distribution is trained on enough data and represented with enough parameters, it begins to look like understanding.

Probability over sequences

We write P(w1, w2, ..., wn) for the joint probability that the exact sequence of n tokens appears. By the chain rule of probability, this joint probability can be factored into a product of conditional probabilities.

P(w_1, w_2, \dots, w_n) = P(w_1) \cdot P(w_2 \mid w_1) \cdot P(w_3 \mid w_1, w_2) \cdots P(w_n \mid w_1, \dots, w_{n-1})

The conditional term P(w3 | w1, w2) asks: after seeing "w1 w2", what is the chance the next token is w3? If we could estimate every such conditional perfectly, we would have a perfect language model. The problem is that there are exponentially many possible histories.

The Markov assumption and n-grams

To make the problem tractable, we make the Markov assumption: a token depends only on the previous n - 1 tokens. This is called an n-gram model. A bigram (2-gram) uses only the previous word; a trigram uses the previous two.

P(w_i \mid w_1, \dots, w_{i-1}) \approx P(w_i \mid w_{i-n+1}, \dots, w_{i-1})

Estimating these probabilities is then just counting in a large corpus. For a bigram, we count how often word B follows word A, divided by how often A appears.

P(w_i \mid w_{i-1}) = \frac{\text{count}(w_{i-1}, w_i)}{\text{count}(w_{i-1})}

from collections import Counter, defaultdict

corpus = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "the cat saw the dog",
]

bigrams = Counter()
unigrams = Counter()
for sentence in corpus:
    tokens = ["<start>"] + sentence.split() + ["<end>"]
    for w1, w2 in zip(tokens, tokens[1:]):
        bigrams[(w1, w2)] += 1
        unigrams[w1] += 1

# probability of "sat" after "cat"
print(bigrams[("cat", "sat")] / unigrams["cat"])

N-gram text generator

Adjust the n-gram order, type a prompt, and generate text. Notice how higher-order n-grams sound more coherent but overfit to the small corpus.

Visualization ngram is coming soon.

Why counting breaks down

N-gram models are easy to understand and fast to train, but they fail for long contexts. The number of possible n-grams grows like V^n, where V is the vocabulary size. Most long contexts never appear in training, so their probability is zero.

Data sparsity: most plausible 5-grams have never been seen.
No generalization: "cat" and "kitten" are not related.
Fixed context: the model cannot look further back when needed.

Smoothing tricks such as add-one (Laplace) smoothing and backoff help, but they are band-aids. The fundamental limit is that the model has no notion of similarity between words; it only memorizes exact strings.

The curse of dimensionality

With a vocabulary of 50,000 words, there are 50,000^3 = 125 trillion possible trigrams. Even the largest text corpora cover only a tiny fraction. This is why modern language models use continuous, learned representations instead of discrete counts.

Perplexity: measuring a language model

Perplexity is the standard evaluation metric for language models. It measures how surprised the model is, on average, by the next token. Lower is better. Mathematically, it is the exponential of the average negative log-likelihood.

\text{perplexity}(W) = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log P(w_i \mid w_{1:i-1})\right)

Intuitively, perplexity is the effective branching factor: if every token were equally likely, perplexity would equal the vocabulary size. A good model assigns most probability mass to the actual next token, so its perplexity is much smaller.

import math

# pretend probabilities the model assigned to the true next tokens
probs = [0.8, 0.3, 0.1, 0.05]
n = len(probs)
log_likelihood = sum(math.log(p) for p in probs) / n
perplexity = math.exp(-log_likelihood)
print(perplexity)

Key takeaway

Language models are probability distributions over token sequences. N-grams make the problem tractable by assuming a short context, but they cannot generalize or capture long-range dependencies. The rest of the course shows how neural networks solve these problems.