Chapter 13: Modern LLMs

The scaling era

After the original transformer, researchers discovered that increasing model size, data, and compute reliably improves performance. This relationship is captured by scaling laws: loss improves predictably with parameters, data, and training flops.

Here N is the number of parameters, D is the number of training tokens, and L_infinity is the loss the model would reach with infinite data and parameters. Alpha and beta are small empirical exponents, roughly 0.08 to 0.1. These laws allowed OpenAI to predict the performance of GPT-3 before training it.

Kaplan et al. 2020 and later Chinchilla from DeepMind refined the relationship. The key insight of Chinchilla is that many models were under-trained: for a fixed compute budget, model size and data size should be scaled together. A smaller model trained on more data can match a larger model trained on less.

GPT, LLaMA, and Mistral

The GPT family is decoder-only and trained on broad internet text. LLaMA showed that smaller, longer-trained models can match larger models. Mistral introduced grouped-query attention and sliding-window attention to improve inference speed without sacrificing quality.

• GPT-4: highly capable, closed weights, widely used via API.
• LLaMA 2/3: open weights, efficient, popular for research and fine-tuning.
• Mistral: strong small models, Mixture-of-Experts variants.

Open-weight models can be downloaded and run locally, which is useful for privacy, customization, and research. Closed models are often easier to use through APIs and may lead on some benchmarks. Both ecosystems are advancing quickly.

Mixture of Experts

A Mixture-of-Experts layer replaces a single feed-forward network with many expert networks. A gating network routes each token to a small subset of experts. This increases model capacity without increasing active parameters per token.

MoE models like Mixtral activate only two out of eight experts per token, making inference faster than a dense model of the same total size while matching or exceeding its performance. The challenge is load balancing: some experts can become overloaded if the routing is not regularized.

Multimodal and tool use

Modern LLMs are not limited to text. Vision-language models process images by slicing them into patches and treating each patch as a token. Audio and video models follow similar ideas. Tool use lets models call external functions, calculators, search engines, and databases.

# example tool use
model = ChatModel(tools=[web_search, calculator])
response = model.chat("What is the population of Jakarta times two?")
# model may emit: {"tool": "web_search", "query": "population of Jakarta"}
# then use calculator on the result

Efficiency and long context

Attention is quadratic in sequence length, which is expensive for long documents. New architectures such as Mamba and RWKV use state-space models and linear attention to scale linearly. RoPE and ALiBi provide better relative position encodings for long contexts.

These alternatives aim to keep the long-range modeling of Transformers while avoiding the memory and compute cost of a full attention matrix. They are an active research area and are starting to appear in production systems alongside classic Transformer blocks.

Open versus closed models

Closed models often lead on benchmarks and are easy to use through APIs. Open-weight models allow customization, privacy, and local deployment. Both ecosystems are advancing rapidly. The choice between them depends on your use case, budget, and whether you need to inspect or modify the weights.