Chapter 11: Training Large Language Models

The pre-training objective

Large language models are pre-trained on a simple but massive task: predict the next token given all previous tokens. This is called causal language modeling. Every web page, book, and code file becomes a training example because each sequence provides many next-token predictions.

The model learns to assign high probability to the actual next token. Because the training data is enormous, the model internalizes grammar, facts, reasoning patterns, and even coding conventions. The objective is the same cross-entropy loss we met in the neural-networks chapters, but applied to every position in a long sequence.

Tokenization at scale

Before training, the text is tokenized into a fixed vocabulary. Byte-Pair Encoding and SentencePiece are the most common algorithms. The vocabulary size is a trade-off: larger vocabularies mean fewer tokens per word but more parameters in the output layer.

# typical tokenization pipeline
text = "Large language models learn from text."
tokens = tokenizer.encode(text)
# tokens: [15496, 40095, 3454, 1021, 11241, ...]
# each token is an integer ID

The tokenizer is trained on a sample of the corpus. It merges frequent pairs until the vocabulary reaches the target size. The same tokenizer is used for training and inference, so vocabulary cannot change later. Data quality matters: documents are deduplicated, filtered for low quality, and sometimes downweighted or removed if they contain too much boilerplate or harmful content.

Data pipeline and packing

Pre-training datasets are measured in terabytes. The data pipeline reads documents, tokenizes them, and packs sequences to a fixed context length. Documents can be concatenated with an end-of-document marker so the model learns boundaries. The goal is to keep every training step fed with high-quality, varied text.

• Deduplication: removes repeated or near-duplicate documents.
• Quality filtering: drops low-quality or machine-generated text.
• Domain mixing: balances web, books, code, and other sources.
• Packing: concatenates short documents to fill the context window.

Optimizers: momentum and Adam

Plain gradient descent updates weights with the current gradient. Adaptive optimizers such as Adam keep moving averages of gradients and squared gradients. This gives per-parameter learning rates and makes training faster and more stable.

Adam uses bias correction at the start of training and divides the update by the square root of v. Modern variants such as AdamW separate weight decay from the gradient, which improves generalization. Most large language models are trained with AdamW and a small weight decay.

Learning rate schedules

The learning rate is usually warmed up from a small value to a peak over a few thousand steps, then decayed with a cosine schedule. Warmup prevents early instability; decay lets the model settle into a flat minimum.

The schedule depends on the total number of training steps T. Some runs decay to a small but non-zero minimum, while others add a short cooldown at the end. Getting the learning rate right is one of the most important hyperparameters.

Batching and parallelism

Training data is packed into batches. For causal language modeling, each sequence in a batch is independent, but the model processes them together for efficiency. Modern training uses data parallelism, model parallelism, and pipeline parallelism to distribute work across thousands of GPUs.

• Data parallelism: each GPU gets a different batch and gradients are averaged.
• Model parallelism: different layers or attention heads live on different GPUs.
• Pipeline parallelism: different micro-batches flow through a pipeline of devices.
• Fully Sharded Data Parallel (FSDP): shards optimizer states and gradients across GPUs to fit larger models.

Monitoring training

Engineers track training loss, validation loss, gradient norms, and learning rate. A rising validation loss while training loss falls is a sign of overfitting. Gradient clipping is used to prevent loss spikes from large gradients.

Training also involves mixed precision, where forward and backward passes use 16-bit or bfloat16 floats to speed up computation and reduce memory, while the optimizer keeps a higher-precision copy of the weights. checkpointing trades computation for memory by recomputing activations during the backward pass instead of storing them.

Overfitting and early stopping

Because pre-training data is huge, modern LLMs rarely overfit in the classical sense. Validation loss may eventually rise as the model memorizes training details, but the more common stopping criterion is simply running out of budget or reaching a target loss. For smaller fine-tuning runs, early stopping on a held-out set is common.