Learn LLM

From base model to assistant

A base model predicts the next token on raw internet text. To make it useful as a chatbot or assistant, we fine-tune it on high-quality instruction and conversation data. This is called supervised fine-tuning, or SFT.

# instruction tuning example
example = {
    "instruction": "Explain photosynthesis in simple terms.",
    "response": "Plants use sunlight, water, and carbon dioxide..."
}
# train the model to produce the response given the instruction

Parameter-efficient fine-tuning

Fine-tuning every parameter of a 70-billion-parameter model is expensive and storage-heavy. Parameter-efficient methods train only a small set of additional parameters while keeping the base model frozen.

LoRA is the most popular technique. It injects low-rank matrices into the attention and feed-forward layers. Instead of updating a large weight matrix W, it learns two small matrices A and B such that the update is BA.

W' = W + BA, \quad B \in \mathbb{R}^{d \times r}, \quad A \in \mathbb{R}^{r \times d}

With rank r much smaller than d, LoRA drastically reduces the number of trainable parameters and makes fine-tuning feasible on consumer hardware.

Reward models and RLHF

Human labelers rank several model outputs by quality. A reward model is trained to predict these rankings. Then the language model is fine-tuned with reinforcement learning to maximize the reward while still producing fluent text.

Collect prompts and several completions per prompt.
Humans rank the completions from best to worst.
Train a reward model on these comparisons.
Fine-tune the language model to maximize reward using PPO.

RLHF helps the model follow instructions, refuse harmful requests, and adopt a helpful tone. However, it also introduces new failure modes: the model may learn to please the reward model rather than be truthful.

DPO and direct preference optimization

Direct Preference Optimization skips the separate reward model. It rephrases the RLHF objective as a classification loss over preference pairs. DPO is simpler, more stable, and often matches PPO in quality.

Alignment trade-offs

Alignment makes models helpful and harmless, but it can also reduce creativity, make answers overly cautious, or encode the values of the labelers. There is active research into constitutional AI, debate, and scalable oversight.

Instruction formats

Different chat models expect different conversation formats. Some use special tokens for user, assistant, and system roles. Using the wrong format can degrade performance dramatically.

<|system|>
You are a helpful assistant.
<|user|>
What is the capital of France?
<|assistant|>
Paris

Key takeaway

Fine-tuning adapts a base model to specific tasks or conversational styles. SFT follows instructions, LoRA makes training affordable, and RLHF/DPO align models with human preferences.