Learn LLM

From base model to assistant

A base model predicts the next token on raw internet text. To make it useful as a chatbot or assistant, we fine-tune it on high-quality instruction and conversation data. This is called supervised fine-tuning, or SFT.

# instruction tuning example
example = {
    "instruction": "Explain photosynthesis in simple terms.",
    "response": "Plants use sunlight, water, and carbon dioxide..."
}
# train the model to produce the response given the instruction

SFT teaches the model the format of conversations, the style of helpful answers, and the kinds of tasks users will ask. It does not usually require as much data as pre-training; a few thousand high-quality examples can be enough.

Parameter-efficient fine-tuning

Fine-tuning every parameter of a 70-billion-parameter model is expensive and storage-heavy. Parameter-efficient methods train only a small set of additional parameters while keeping the base model frozen.

LoRA is the most popular technique. It injects low-rank matrices into the attention and feed-forward layers. Instead of updating a large weight matrix W, it learns two small matrices A and B such that the update is BA.

W' = W + BA, \quad B \in \mathbb{R}^{d \times r}, \quad A \in \mathbb{R}^{r \times d}

With rank r much smaller than d, LoRA drastically reduces the number of trainable parameters and makes fine-tuning feasible on consumer hardware. A rank of 4, 8, or 16 is common; the best rank depends on the task and the amount of data.

LoRA parameter savings

Drag the rank slider and see how many parameters LoRA trains compared to a full fine-tune of the same weight matrix.

LoRA parameter savings

W' = W + BA, where B is d × r and A is r × d

Matrix dimension d = 64

LoRA rank r = 8

4,096

Full fine-tune params

1,024

LoRA trainable params

25.0% of full fine-tune

The tiny grid is a stylized view of the d × d update matrix. Low rank means the update is constrained to a small subspace, saving parameters.

QLoRA and quantization

QLoRA goes further by quantizing the base model to 4-bit precision while keeping the LoRA weights in higher precision. This fits very large models into limited GPU memory without a large accuracy drop. The 4-bit weights are dequantized on the fly for the forward pass, and only the small LoRA matrices are updated.

Reward models and RLHF

Human labelers rank several model outputs by quality. A reward model is trained to predict these rankings. Then the language model is fine-tuned with reinforcement learning to maximize the reward while still producing fluent text.

Collect prompts and several completions per prompt.
Humans rank the completions from best to worst.
Train a reward model on these comparisons.
Fine-tune the language model to maximize reward using PPO.

RLHF helps the model follow instructions, refuse harmful requests, and adopt a helpful tone. However, it also introduces new failure modes: the model may learn to please the reward model rather than be truthful.

DPO and direct preference optimization

Direct Preference Optimization skips the separate reward model. It rephrases the RLHF objective as a classification loss over preference pairs. DPO is simpler, more stable, and often matches PPO in quality. It has become a popular alternative for aligning open-weight models.

\mathcal{L}_{\text{DPO}} = -\log \sigma\left(\beta \log \frac{\pi(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)

Here x is the prompt, y_w is the preferred completion, y_l is the dispreferred completion, pi_ref is the base policy, and beta controls how far the fine-tuned model can drift from the base. DPO directly optimizes the model's preference probability without a reward network.

Instruction formats

Different chat models expect different conversation formats. Some use special tokens for user, assistant, and system roles. Using the wrong format can degrade performance dramatically.

<|system|>
You are a helpful assistant.
<|user|>
What is the capital of France?
<|assistant|>
Paris

Alignment trade-offs

Alignment makes models helpful and harmless, but it can also reduce creativity, make answers overly cautious, or encode the values of the labelers. There is active research into constitutional AI, debate, and scalable oversight.

Key takeaway

Fine-tuning adapts a base model to specific tasks or conversational styles. SFT follows instructions, LoRA makes training affordable, and RLHF/DPO align models with human preferences. Instruction formats and alignment trade-offs are practical details that matter for real deployments.

What comes next

A fine-tuned model is only useful if you can evaluate and deploy it well. The final chapter covers prompting, retrieval, benchmarks, failure modes, and the engineering of real-world LLM applications.