Learn LLM

Why we need vectors for words

In an n-gram model, "cat" and "kitten" are completely unrelated symbols. A neural language model instead represents every word as a dense vector in a high-dimensional space. Words that appear in similar contexts end up near each other in that space.

These vectors are called embeddings. They encode meaning as geometry. Similar words cluster together, and analogies such as king - man + woman ≈ queen appear as vector arithmetic.

One-hot versus dense vectors

A naive representation is a one-hot vector: a 50,000-dimensional vector with a single 1 at the vocabulary index. It is huge, sparse, and gives every word the same distance from every other word.

e_{\text{cat}} = [0,0,1,\dots,0], \quad e_{\text{kitten}} = [0,1,0,\dots,0]

An embedding matrix E of size V × d maps each word index to a dense d-dimensional vector. During training, the model learns E so that words with similar meanings or contexts have similar vectors.

v_w = E_w \in \mathbb{R}^d

Learning embeddings from context

Word2Vec showed that you can learn good embeddings with a simple objective: predict a word from its neighbors (CBOW) or predict its neighbors from the word (skip-gram). The model learns to place words with shared contexts close together.

# simplified skip-gram training objective
# for context window size 2 around target word w_t
loss = -sum(log P(w_{t+j} | w_t) for j in [-2, -1, 1, 2])

The softmax over the full vocabulary is expensive, so Word2Vec uses negative sampling. Instead of normalizing over all words, it only scores the true context words and a few random negative examples.

Subword tokenization

No vocabulary can list every word. Proper names, typos, and rare technical terms appear constantly. Subword tokenization splits text into frequent pieces called tokens. A word can be one token or several, so the model can represent unknown words from their parts.

Byte-Pair Encoding tokenizer

Type text and watch the BPE algorithm merge frequent character pairs. The same vocabulary can represent common words with one token and rare words with several.

Visualization tokenizer is coming soon.

Byte-Pair Encoding starts with every character as a token and repeatedly merges the most frequent adjacent pair. The result is a vocabulary that balances compactness and expressiveness.

corpus = ["low", "lower", "lowest", "newer", "newest"]
# initial tokens: l o w </w>, n e w </w>, etc.
# merge 'lo' because it appears in low, lower, lowest
# then 'ne' because it appears in newer, newest
# continue until vocabulary size target is reached

Positional information

Word embeddings do not know where a word appears in a sentence. A transformer adds positional encoding to each token embedding, either as learned vectors or as fixed sinusoidal patterns.

PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d}}\right), \quad PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d}}\right)

The sinusoidal pattern allows the model to generalize to positions it never saw during training. Each dimension oscillates at a different frequency, creating a unique fingerprint for every position.

Tokens, not words

From now on, we use token instead of word. A token can be a whole word, a piece of a word, or even a single punctuation mark. The model sees a sequence of token IDs and learns an embedding for each.

Key takeaway

Embeddings turn discrete tokens into continuous vectors that capture meaning through context. Subword tokenizers let the model handle rare and unknown words by breaking them into familiar pieces.