From tokens to vectors
A neural network cannot multiply a word. Before a token reaches the first weight matrix, it must be turned into a vector of numbers. The simplest vector is one-hot: a 50,000-dimensional entry for every word, with a single 1 and 49,999 zeros. One-hot vectors are easy to understand, but they treat every word as unrelated. 'cat' and 'kitten' have the same distance as 'cat' and 'banana'.
Embeddings solve this by learning a dense vector for each token. Instead of 50,000 dimensions, each token gets 128 or 512 continuous values. The network learns to place similar tokens close together and unrelated tokens far apart. The geometry of the embedding space starts to match our intuition about meaning.
One-hot versus dense vectors
A one-hot vector is a lookup table that uses the identity matrix. Each dimension is a word, and only one dimension is active. Because every pair of one-hot vectors is orthogonal, the dot product between any two different words is zero.
An embedding matrix E of size V x d maps a vocabulary index to a dense d-dimensional vector. The lookup is just selecting a row of the matrix. During training, the values in E are updated by backpropagation, just like any other weight matrix.
Learning embeddings from context
Word2Vec showed that high-quality embeddings can be learned from a simple idea: words that appear in similar contexts have similar meanings. Two training objectives dominate. CBOW predicts a target word from its surrounding context, and skip-gram predicts the context words from the target.
# simplified skip-gram objective for window size 2
loss = -sum(log P(w_{t+j} | w_t) for j in [-2, -1, 1, 2])The softmax over the entire vocabulary is expensive, so Word2Vec uses negative sampling. The model only scores the true context words and a handful of random negative examples. This makes training fast and scales to billions of words.
GloVe takes a different approach and factorizes global word co-occurrence statistics. FastText extends Word2Vec by representing words as bags of character n-grams, which helps it handle rare words and morphological variants.
Analogies and vector arithmetic
Because embeddings live in a continuous space, relationships become directions. The classic example is king - man + woman ~ queen. The vector from man to woman captures a gender direction, and applying that direction to king moves it to queen.
This works because the training objective pushes words with similar contexts close together and organizes unrelated concepts along consistent axes. Cosine similarity is the standard way to compare two embedding vectors.
Subword tokenization
No vocabulary can list every word. Proper names, typos, rare technical terms, and morphological variants appear constantly in real text. Subword tokenization splits text into frequent pieces called tokens. A word can be one token or several, so the model can represent unknown words from their parts.
Byte-Pair Encoding, or BPE, is one of the most widely used subword algorithms. It starts with every character as its own token plus an end-of-word marker. It then counts every adjacent pair of tokens in the training corpus and merges the most frequent pair into a new single token. The process repeats until the vocabulary reaches a target size.
The algorithm is greedy and data-driven. If the characters l and o appear next to each other often, BPE creates a lo token. If low and est appear next to each other often, it creates a lowest token. The vocabulary stays compact because rare words are split into smaller pieces, while common words become single tokens.
Modern systems such as GPT-2, GPT-4, and LLaMA use BPE with additional rules for spaces, Unicode, and special tokens. Some variants merge raw bytes instead of characters so that every Unicode symbol can be represented. The result is an open vocabulary: any text can be tokenized, even if it was never seen during training.
corpus = ["low", "lower", "lowest", "newer", "newest"]
# pre-tokenize: l o w </w>, n e w </w>, etc.
# count all adjacent pairs, e.g. (l, o), (o, w), (w, </w>), (n, e), ...
# merge the most frequent pair, e.g. (l, o) -> lo
# repeat until the vocabulary size target is reachedStatic and contextual embeddings
Word2Vec and GloVe produce static embeddings: each token has one vector no matter where it appears. But words are ambiguous. 'Bank' can mean a river edge or a financial institution. Static embeddings average these senses into one vector.
Contextual embeddings, introduced by ELMo and later BERT, produce a different vector for a token based on the sentence around it. The same word can have different vectors in different contexts. These models are the direct ancestors of modern large language models, and we explore them in later chapters.
Positional information
Word embeddings do not know where a word appears in a sentence. A transformer adds positional encoding to each token embedding, either as learned vectors or as fixed sinusoidal patterns.
The sinusoidal pattern allows the model to generalize to positions it never saw during training. Each dimension oscillates at a different frequency, creating a unique fingerprint for every position.
Tokens, not words
From now on, we use token instead of word. A token can be a whole word, a piece of a word, or even a single punctuation mark. The model sees a sequence of token IDs and learns an embedding for each.
Key takeaway
Embeddings turn discrete tokens into continuous vectors that capture meaning through context. Subword tokenizers let the model handle rare and unknown words by breaking them into familiar pieces. Contextual embeddings go one step further and produce a different vector for the same token in different sentences.