Learn LLM

From trained model to text

After pre-training, a language model can generate text by sampling one token at a time. The chosen token is appended to the context, and the model runs again to predict the next one. Repeating this produces a sequence.

def generate(prompt, max_length=50):
    tokens = tokenizer.encode(prompt)
    for _ in range(max_length):
        logits = model(tokens)
        probs = softmax(logits[-1])
        next_token = sample(probs)
        tokens.append(next_token)
    return tokenizer.decode(tokens)

Greedy and beam search

Greedy decoding always picks the token with highest probability. It is deterministic but often repetitive and boring. Beam search keeps the top k partial sequences and selects the one with the best overall log-probability.

\text{score}(w_{1:t}) = \sum_{i=1}^{t} \log P(w_i \mid w_{1:i-1})

Beam search reduces obvious mistakes, but it still tends to produce generic and repetitive text. Language models generate more creative and natural output when randomness is introduced.

Sampling with temperature

Temperature scales the logits before softmax. A low temperature makes the distribution sharp, so the model almost always picks the top token. A high temperature flattens the distribution, allowing less likely but more surprising tokens.

P(w) = \frac{\exp(z_w / T)}{\sum_{v} \exp(z_v / T)}

Sampling lab

Adjust temperature, top-k, and top-p to see how the probability distribution changes. Generate tokens and observe the trade-off between coherence and creativity.

Visualization sampling is coming soon.

Top-k and nucleus sampling

Temperature alone sometimes allows very unlikely tokens, producing nonsense. Top-k sampling restricts the choice to the k most likely tokens. Nucleus or top-p sampling is more flexible: it keeps the smallest set of tokens whose cumulative probability exceeds p.

For example, with top-p = 0.9, we sample only from the tokens that together account for 90% of the probability mass. If the model is confident, this may be a single token; if it is uncertain, it may be many.

def top_p_sampling(logits, p=0.9):
    probs = softmax(logits)
    sorted_probs = np.sort(probs)[::-1]
    sorted_indices = np.argsort(probs)[::-1]
    cumsum = np.cumsum(sorted_probs)
    cutoff = np.searchsorted(cumsum, p) + 1
    kept = sorted_indices[:cutoff]
    probs = probs[kept] / probs[kept].sum()
    return np.random.choice(kept, p=probs)

Repetition and length penalties

Without intervention, models repeat phrases. Repetition penalties subtract from the logits of tokens that have already appeared. Length penalties encourage the model to stop at appropriate points. Stop sequences can also be used to end generation cleanly.

Temperature is not intelligence

Changing temperature does not make the model more or less knowledgeable. It only controls randomness. A low temperature gives predictable but possibly dull text; a high temperature gives varied but possibly incoherent text.

Key takeaway

Generation is repeated next-token sampling. Greedy and beam search are deterministic; temperature, top-k, and top-p add randomness. The right settings depend on whether you want coherent, creative, or constrained output.