Learn LLM

The scaling era

After the original transformer, researchers discovered that increasing model size, data, and compute reliably improves performance. This relationship is captured by scaling laws: loss improves predictably with parameters, data, and training flops.

L(N, D) \propto \frac{1}{N^{0.076}} + \frac{1}{D^{0.095}}

Here N is the number of parameters and D is the amount of training data. These empirical laws allowed OpenAI to predict the performance of GPT-3 before training it.

GPT, LLaMA, and Mistral

The GPT family is decoder-only and trained on broad internet text. LLaMA showed that smaller, longer-trained models can match larger models. Mistral introduced grouped-query attention and sliding-window attention to improve inference speed without sacrificing quality.

GPT-4: highly capable, closed weights, widely used via API.
LLaMA 2/3: open weights, efficient, popular for research and fine-tuning.
Mistral: strong small models, Mixture-of-Experts variants.

Mixture of Experts

A Mixture-of-Experts layer replaces a single feed-forward network with many expert networks. A gating network routes each token to a small subset of experts. This increases model capacity without increasing active parameters per token.

\text{MoE}(x) = \sum_{i=1}^{E} g(x)_i \cdot E_i(x)

MoE models like Mixtral activate only two out of eight experts per token, making inference faster than a dense model of the same total size while matching or exceeding its performance.

Multimodal and tool use

Modern LLMs are not limited to text. Vision-language models process images by slicing them into patches and treating each patch as a token. Audio and video models follow similar ideas. Tool use lets models call external functions, calculators, search engines, and databases.

# example tool use
model = ChatModel(tools=[web_search, calculator])
response = model.chat("What is the population of Jakarta times two?")
# model may emit: {"tool": "web_search", "query": "population of Jakarta"}
# then use calculator on the result

Efficiency and long context

Attention is quadratic in sequence length, which is expensive for long documents. New architectures such as Mamba and RWKV use state-space models and linear attention to scale linearly. RoPE and ALiBi provide better relative position encodings for long contexts.

The open vs closed debate

Closed models often lead on benchmarks and are easy to use through APIs. Open-weight models allow customization, privacy, and local deployment. Both ecosystems are advancing rapidly.

Key takeaway

Modern LLMs are built on scaling laws, efficient attention, MoE layers, and multimodal extensions. They are available as both closed APIs and open weights, and research continues on long-context and sub-quadratic architectures.