What is a language model?
A language model is, at its core, a probability distribution over sequences of words or tokens. Given a sequence like "the cat sat on the", a language model assigns a probability to every possible next token: "mat", "floor", "moon", or even "banana". The ones that sound more plausible get higher probability.
This simple idea is the engine behind auto-complete, machine translation, speech recognition, and modern chatbots. If we can estimate P(next token | previous tokens) well, we can generate coherent text by repeatedly sampling the most likely next token.
From probabilities to text
A language model does not understand meaning the way humans do. It captures statistical patterns: which words tend to follow which contexts. Surprisingly, when this distribution is trained on enough data and represented with enough parameters, it begins to look like understanding.
Probability over sequences
We write P(w1, w2, ..., wn) for the joint probability that the exact sequence of n tokens appears. By the chain rule of probability, this joint probability can be factored into a product of conditional probabilities.
The conditional term P(w3 | w1, w2) asks: after seeing "w1 w2", what is the chance the next token is w3? If we could estimate every such conditional perfectly, we would have a perfect language model. The problem is that there are exponentially many possible histories.
The Markov assumption and n-grams
The full history of a sentence can be arbitrarily long, making it impractical to estimate a separate probability for every possible history. The Markov assumption simplifies the problem by saying that, once we know a fixed number of recent tokens, earlier tokens provide no additional information about the next token. In other words, the model has a limited memory.
For an n-gram model, that limited memory contains the previous n - 1 tokens. A bigram (2-gram) predicts using one previous token, while a trigram (3-gram) uses two. For example, when predicting the next word after "the cat sat", a trigram model considers only "cat sat" and behaves as if anything before those words is irrelevant.
An approximation, not a fact about language
Natural language does not truly obey the Markov assumption: a word can depend on something written many sentences earlier. The assumption deliberately trades long-range information for a much smaller number of contexts that can be counted and estimated from data. Increasing n preserves more context, but also makes each exact context rarer.
Estimating these probabilities is then just counting in a large corpus. For a bigram, we count how often word B follows word A, divided by how often A appears.
from collections import Counter, defaultdict
corpus = [
"the cat sat on the mat",
"the dog sat on the log",
"the cat saw the dog",
]
bigrams = Counter()
unigrams = Counter()
for sentence in corpus:
tokens = ["<start>"] + sentence.split() + ["<end>"]
for w1, w2 in zip(tokens, tokens[1:]):
bigrams[(w1, w2)] += 1
unigrams[w1] += 1
# probability of "sat" after "cat"
print(bigrams[("cat", "sat")] / unigrams["cat"])Bag of words
N-grams predict the next token by remembering a short prefix. A different way to use counting is to represent an entire document as a multiset of its words, ignoring grammar and word order. This is called a bag-of-words model. It is the simplest way to turn text into a numerical vector.
In this model, each dimension of the vector corresponds to one word in the vocabulary. The value is the count (or binary presence) of that word in the document. Two sentences like "the cat sat on the mat" and "the mat sat on the cat" become identical vectors, even though their meanings differ.
from collections import Counter
corpus = [
"the cat sat on the mat",
"the dog sat on the log",
"the cat saw the dog",
]
# build vocabulary
vocab = sorted({word for sentence in corpus for word in sentence.split()})
print("Vocabulary:", vocab)
# represent each document as a count vector
for sentence in corpus:
counts = Counter(sentence.split())
vector = [counts[word] for word in vocab]
print(sentence, "->", vector)Order disappears
Bag-of-words captures which words are present and how often, but it discards all information about their sequence. This makes it fast and compact for document-level tasks such as search and classification, but useless for modeling syntax or generating coherent text.
Because the vector is mostly zeros, we call it a sparse vector. Each non-zero entry is a real word with an interpretable count. In the next chapter we will see how TF-IDF improves on raw counts, and later how dense word embeddings overcome the limitation of having no notion of similarity between words.
Why counting breaks down
N-gram and bag-of-words models are easy to understand and fast to train, but they share the same weakness: they are built from exact discrete counts. N-grams fail for long contexts because the number of possible n-grams grows like V^n, where V is the vocabulary size. Most long contexts never appear in training, so their probability is zero. Bag-of-words suffers too: it ignores word order entirely, so "dog bites man" and "man bites dog" look the same.
- Data sparsity: most plausible 5-grams have never been seen.
- No generalization: "cat" and "kitten" are not related.
- Lost order: bag-of-words cannot distinguish "the cat sat" from "the sat cat".
- Fixed context: n-gram models cannot look further back when needed.
Smoothing tricks such as add-one (Laplace) smoothing and backoff help, but they are band-aids. The fundamental limit is that the model has no notion of similarity between words; it only memorizes exact strings.
The curse of dimensionality
With a vocabulary of 50,000 words, there are 50,000^3 = 125 trillion possible trigrams. Even the largest text corpora cover only a tiny fraction. This is why modern language models use continuous, learned representations instead of discrete counts.
Perplexity: the number a language model is trained against
So far we have a model that assigns probabilities to sequences. How do we know it is any good? We need one number that summarizes how well the model predicts text. That number is perplexity.
Perplexity measures how surprised the model is, on average, by the next token. Lower is better. Mathematically, it is the exponential of the average negative log-likelihood over the whole sequence.
Intuitively, perplexity is the effective branching factor: if every token were equally likely, perplexity would equal the vocabulary size. A good model assigns most probability mass to the actual next token, so its perplexity is much smaller. A perplexity of 20 means the model is as confused as if it had to choose among 20 equally likely tokens at every step.
The formula conditions each token on all previous tokens, so perplexity can only be computed for models that define a real next-token distribution. N-grams fit naturally: a trigram approximates P(w_i | w_{1:i-1}) with P(w_i | w_{i-2}, w_{i-1}). Bag-of-words, by design, has no memory of order and cannot predict the next token. If forced to do so, it would produce the same flat distribution for every context, giving a perplexity close to the vocabulary size. Perplexity thus makes the difference concrete: n-grams are weak language models, while bag-of-words is not a sequence model at all — it is a document representation.
Perplexity is just cross-entropy loss in disguise
Perplexity equals exp(cross-entropy loss). The loss that gradient descent minimizes during training is the same number, just on a log scale. So when you see a training loss curve going down, you are watching perplexity shrink.
This is why the metric matters beyond evaluation: it is the objective. Every language model, from a trigram to a modern LLM, is trained to reduce this exact number. You will meet it again in two places later in the course: it is the quantity plotted on the Y-axis of training loss curves in Chapter 10, and it is the L in the scaling laws of Chapter 12, which predicts how loss falls as models get bigger and data increases.
import math
# pretend probabilities the model assigned to the true next tokens
probs = [0.8, 0.3, 0.1, 0.05]
n = len(probs)
log_likelihood = sum(math.log(p) for p in probs) / n
perplexity = math.exp(-log_likelihood)
print(perplexity)Do modern LLMs still use perplexity?
Yes, but only on the training side. Perplexity is still the objective every LLM is optimized against, so it is what researchers report during pretraining. For comparing assistants, it has been replaced by benchmarks like MMLU and HumanEval: a model can have great perplexity and still give bad answers, because predicting text well is not the same as being helpful. Perplexity tells you how fluently a model speaks; benchmarks tell you whether it knows things.
Key takeaway
Language models are probability distributions over token sequences. N-grams make the problem tractable by assuming a short context, bag-of-words gives a simple document representation, and neither can generalize or capture long-range dependencies. The rest of the course shows how neural networks solve these problems.