Learn LLM

The pre-training objective

Large language models are pre-trained on a simple but massive task: predict the next token given all previous tokens. This is called causal language modeling. Every web page, book, and code file becomes a training example because each sequence provides many next-token predictions.

\mathcal{L} = -\sum_{t=1}^{T} \log P(w_t \mid w_{1:t-1})

The model learns to assign high probability to the actual next token. Because the training data is enormous, the model internalizes grammar, facts, reasoning patterns, and even coding conventions.

Tokenization at scale

Before training, the text is tokenized into a fixed vocabulary. Byte-Pair Encoding and SentencePiece are the most common algorithms. The vocabulary size is a trade-off: larger vocabularies mean fewer tokens per word but more parameters in the output layer.

# typical tokenization pipeline
text = "Large language models learn from text."
tokens = tokenizer.encode(text)
# tokens: [15496, 40095, 3454, 1021, 11241, ...]
# each token is an integer ID

The tokenizer is trained on a sample of the corpus. It merges frequent pairs until the vocabulary reaches the target size. The same tokenizer is used for training and inference, so vocabulary cannot change later.

Optimizers: momentum and Adam

Plain gradient descent updates weights with the current gradient. Adaptive optimizers such as Adam keep moving averages of gradients and squared gradients. This gives per-parameter learning rates and makes training faster and more stable.

m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t, \quad v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2

Adam uses bias correction at the start of training and divides the update by the square root of v. Modern variants such as AdamW separate weight decay from the gradient, which improves generalization.

Learning rate schedules

The learning rate is usually warmed up from a small value to a peak over a few thousand steps, then decayed with a cosine schedule. Warmup prevents early instability; decay lets the model settle into a flat minimum.

Training loss curve

Adjust the learning rate and batch size to see how loss behaves. Unstable rates produce spikes; too-small rates learn too slowly.

Training & Validation Loss

Train loss

Val loss

Batching and parallelism

Training data is packed into batches. For causal language modeling, each sequence in a batch is independent, but the model processes them together for efficiency. Modern training uses data parallelism, model parallelism, and pipeline parallelism to distribute work across thousands of GPUs.

Compute cost

Training a 7-billion-parameter model can cost hundreds of thousands of dollars in GPU time. A 175-billion-parameter model can cost millions. This is why pre-training is usually done by well-funded labs, while fine-tuning is more accessible.

Monitoring training

Engineers track training loss, validation loss, gradient norms, and learning rate. A rising validation loss while training loss falls is a sign of overfitting. Gradient clipping is used to prevent loss spikes from large gradients.

g \leftarrow \min\left(1, \frac{\tau}{||g||}\right) g

Key takeaway

Pre-training is next-token prediction on massive text corpora. It requires a tokenizer, an optimizer like AdamW, a learning rate schedule, and large-scale distributed compute. The result is a base model that can later be fine-tuned.