Chapter 2: TF-IDF & Sparse Retrieval

Beyond raw counting

The simplest way to represent text is to count what is in it. N-gram models count sequences of tokens, and a bag-of-words model counts how many times each word appears in a document. These methods are easy to understand, fast to compute, and surprisingly useful.

But counting has a serious blind spot: it only recognizes exact matches. The sentences "the dog bites" and "the hound nips" describe the same event, yet they share no words except "the". A count-based system gives them zero similarity, just as it would for "the dog bites" and "the banana sings".

TF-IDF is a smarter way to count. Instead of treating every word as equally important, it assigns a weight to each term based on how often the term appears in a document and how rare it is across the whole collection. This gives us a numerical representation of a document, but it is still built from discrete word counts rather than learned meaning.

Term Frequency (TF)

Term frequency is the first ingredient. It tells us how prominent a word is inside a particular document. If a document repeats the word "neural" many times, that word is probably a good indicator of what the document is about.

In this formula, t is a term and d is a document. The numerator counts how many times t occurs in d, and the denominator normalizes by the document length. A 100-word document that contains "transformer" ten times gets the same TF score as a 1,000-word document that contains the word one hundred times.

There are other definitions of TF. Some use the raw count, some use the logarithm of the count, and some cap the value at one to dampen the effect of very frequent terms. For our purposes, normalized frequency is the most intuitive starting point.

Inverse Document Frequency (IDF)

Term frequency alone is not enough. Common words like "the", "and", and "is" appear in almost every document, so a high TF for these words does not tell us much about a document's topic. We need a way to downweight words that are common everywhere and upweight words that are rare and distinctive.

Inverse document frequency captures exactly that idea. It looks at the whole collection of documents and asks: how many of them contain this term? If the term appears in every document, its IDF is near zero. If it appears in only one document out of thousands, its IDF is high.

Here N is the total number of documents in the collection, and df(t) is the number of those documents that contain the term t. The logarithm turns the ratio into a more manageable scale: a word that appears in one out of a thousand documents is interesting, but not a thousand times more interesting than a word that appears everywhere.

For example, imagine a small collection of science articles. The word "the" appears in every document, so its IDF is log(N / N) = 0. The word "transformer" appears in only a few documents, so its IDF is high. That is exactly what we want: rare, informative words get larger weights.

TF-IDF = TF × IDF

TF-IDF combines the two ideas into a single score. A term gets a high TF-IDF value when it appears often in the current document and rarely in the rest of the collection. This means it is a strong signal for what makes this document unique.

Think of it as a simple signal-to-noise ratio. TF is the signal: how loud is this word inside the document? IDF is the noise filter: how common is this word in the background of all documents? The product keeps words that are both locally frequent and globally rare.

Here is a concrete example using scikit-learn. We create a tiny corpus, vectorize the documents, and print the resulting TF-IDF matrix. The numbers in the matrix are the TF-IDF weights for each term in each document.

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "the cat sat on the mat",
    "the dog barked at the mail carrier",
    "the transformer revolutionized natural language processing",
    "neural networks learn from data",
]

vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)

print("Feature names:", vectorizer.get_feature_names_out())
print("TF-IDF matrix (dense):")
print(X.toarray().round(3))

Run the code and look at the row for the "transformer" sentence. The words "transformer", "revolutionized", "processing", and "language" should have larger weights than common words like "the" and "from". That is TF-IDF doing what it was designed to do.

The sparse vector problem

The output of TfidfVectorizer is a matrix whose width equals the size of the vocabulary. For a real corpus that can easily be 50,000 dimensions or more. Most entries in each row are zero, because a single document only contains a tiny fraction of the full vocabulary.

This kind of representation is called a sparse vector. It is efficient to store, and it is easy to interpret: each dimension is a real word, and the value is a weight. But its geometry is impoverished. Two documents about the same topic will still have zero overlap if they use different words to describe it.

Worse, the similarity between any two distinct words is always the same. "car" and "automobile" occupy completely different dimensions, so their cosine similarity is zero. "car" and "banana" also have zero cosine similarity, even though one pair is clearly related and the other is not.

That limitation is why TF-IDF is only a stepping stone. To capture meaning, we need dense vectors where similar words live near each other. Neural networks learn exactly those kinds of continuous representations: instead of counting discrete words, they adjust real-valued weights until similar words end up close together. In the next chapter we build one from the smallest unit, the perceptron, and watch how stacking neurons creates rich, non-linear representations. Later we will see how those same ideas produce dense word embeddings in Chapter 7.