LLM Basics 5 — Generating Text

Series: 1 · What a model is on disk → 2 · Tokens and embeddings → 3 · A layer is matrix multiplication → 4 · The forward pass → 5 · Generating text → 6 · Attention and the KV cache → 7 · The hardware fit → Capstone · Why-LLM-Decode-Is-Memory-Bound

Lesson 4 ended with the model’s entire output: one forward pass — a complete run through the model — produces a probability for every one of the 32,256 tokens in its vocabulary, estimating which comes next. That is a list of numbers, not text. This lesson turns it into text, and explains the settings — temperature, top-k, top-p — that every LLM tool exposes.

The loop

Text comes from running the model over and over:

int tokens[MAX];  int n = tokenize(prompt, tokens);    // lesson 2

for (;;) {
    float probs[32256];
    forward_pass(tokens, n, probs);   // lesson 4: one distribution for the next token
    int next = choose(probs);         // this lesson: pick one
    if (next == EOS || n == MAX) break;
    tokens[n++] = next;               // append it, and go again
    print_token(next);
}

Pick a token, append it, run again. Each new token becomes part of the input for the next. This is what autoregressive means: the model’s output is fed back in as its own input. It is also why generation is slow. Each token needs its own forward pass, and a token can’t be computed until the one before it has been chosen (lesson 4 measured this as decode, at 33 tokens/s, against roughly 2,400 tokens/s for prefill).

Watching it run

Observed: the real model, continuing def fibonacci(n):⏎ if n by always taking the most likely token. Each row is one trip around the loop. ⏎ is a newline, and · marks spaces where they are hard to see.

Result, after 24 trips:

def fibonacci(n):
    if n == 0:
        return 0
    elif n == 1:
        return 1
    else:

Three things to notice:

• The first choice was the hard one. At step 1 the model was nearly split between ·==· (31%) and ·<=· (28%).
• Each choice constrains the next. Once if n == 0: was written, the rest of the structure was almost forced: most later steps are 0.97–1.00. The loop commits. A different first token would have led to a different, equally valid program, and an unlucky early token can lead somewhere bad.
• Indentation is tokens too. Step 5 is seven spaces as one token, and the eighth space is glued to ·return (lesson 2).

Stopping

The loop above stopped because it hit a length limit of 24 tokens. The other way it stops is by choice.

The vocabulary contains special tokens that are not text. The important one is end-of-sequence (EOS): a token the model learned, during training, to produce when a response is finished. When the loop picks it, generation ends.

• Observed: this model’s EOS token is <|EOT|>, ID 32021, recorded in the GGUF file’s metadata (tokenizer.ggml.eos_token_id).
• Observed: asked to write a function that squares a number, the model wrote it, added a sentence of explanation, and then chose EOS by itself after 68 tokens. The server reported stop_type: eos.

Tools usually add two more ways to stop: a maximum token count, and stop strings — text that, once generated, ends the loop.

Choosing: greedy or sampling

choose(probs) is where all the settings live. There are two basic strategies.

• Greedy always takes the single most likely token, as in the run above.
  • It is deterministic: the same prompt always gives the same text.
  • It tends toward bland and repetitive text, because it never takes a less likely but better path.
• Sampling picks at random, weighted by the probabilities. A token with probability 0.31 is chosen 31% of the time.
  • In C terms, it draws a random number r between 0 and 1 and walks the cumulative sum of probabilities until it passes r.
  • The randomness comes from a pseudo-random number generator (PRNG), and its starting value is the seed. The same seed gives the same “random” choices.

Observed: five runs of the same prompt, sampled with different seeds:

Seed 3, run twice more, produced exactly the same text both times.

Five seeds produced five different programs:

• Seed 4 took the 12% branch — n < 0 — and continued with what a negative-input check usually contains: an error message.
• That message is in Chinese: “the input number must be a positive number”. The model was trained on code with comments and messages in many languages. Taking a less likely first token led it into a less common part of what it learned.

The settings, and what each does to the list

Every common setting is a rule for reshaping the probability list before sampling from it. Here they are applied to the real distribution for the first token above.

Observed (llama.cpp reports the probabilities after each adjustment):

Temperature — sharpen or flatten

Temperature divides every logit by a number T before the softmax. (A logit is the model’s raw score for a token, and softmax turns logits into probabilities — both from lesson 4.)

• T below 1 sharpens. Gaps between logits grow, so likely tokens get likelier.
  • At 0.5, ·==· rose from 31% to 43%, and ·in fell from 2.4% to 0.3%.
• T above 1 flattens. Gaps shrink, so unlikely tokens get a better chance.
  • At 1.5, the top token fell to 24%.
• T = 0 is greedy. All the probability goes to the top token. That is why “temperature 0” is the usual way to ask for deterministic output.

The arithmetic: dividing logits by T is the same as raising each probability to the power 1/T and rescaling so they add to 1. At T = 0.5, 0.307² = 0.094, and divided by the new total (about 0.219), that is 0.431 — the table’s value.

Top-k — keep the k most likely

Top-k keeps only the k most probable tokens, discards the rest, and rescales the survivors to add to 1.

• With k = 3, the three survivors had 0.307 + 0.279 + 0.145 = 0.731, so each was divided by 0.731. The top one becomes 0.307 / 0.731 = 0.420.
• The weakness: k is fixed. When the model is confident, 3 is too many; when many tokens are reasonable, 3 is too few.

Top-p (nucleus sampling) — keep enough to cover p

Top-p keeps the smallest group of top tokens whose probabilities add up to at least p.

• With p = 0.8: the top three add to 0.731, which falls short, and adding ·<· reaches 0.853. So 4 tokens stay.
• The group adapts: when the model is confident, one or two tokens cover 80%; when it is unsure, many do.

Min-p — keep anything close to the best

Min-p keeps every token whose probability is at least min_p times the top token’s.

• With min-p 0.3, the cut-off is 0.3 × 0.307 = 0.092. Five tokens clear it, and ·in, at 0.024, does not.
• It also adapts: a confident model has a high top probability and so a high cut-off, which prunes hard.

In practice

• Engines apply these as a chain, in a set order, and their defaults differ. Check your engine’s documentation rather than assuming.
• Low temperature, or greedy, suits code and factual answers. There is usually one right continuation, and creativity mostly adds errors.
• Higher temperature suits brainstorming and prose, where variety is the point.
• None of these settings make the model know more. They only decide how adventurously to pick from what it already predicted.

Chat is the same loop

A chat interface does not use a different mechanism. The conversation is written out as one long piece of text, in a format the model was trained on — a chat template — and the loop continues it.

• Observed: this model answered and stopped cleanly when its prompt used the ### Instruction: / ### Response: layout that DeepSeek-Coder’s instruct models use. The squaring-function example above came from it.
• A template is a convention, so a wrong one quietly degrades the output.
  • Observed: this particular GGUF file carries no template; its metadata has no tokenizer.chat_template entry.
  • Observed: asked to format a chat anyway, llama.cpp’s server produced the ChatML layout (<|im_start|>system …), whose markers this model’s tokenizer splits into ordinary pieces: <, |, im, _, start, |, >.
  • So the model would see unfamiliar punctuation where it expects structure. It would still respond, but worse, and with no error to tell you why.
  • Newer model files embed their template, and engines read it from there.

What each trip around the loop costs

Every generated token is one forward pass, and every forward pass reads all of the model’s weights from memory (lessons 1 and 4). That is why generation runs at tens of tokens per second while reading a prompt runs at thousands, and it is the problem the capstone examines — including speculative decoding, which checks several guessed tokens in one pass.

There is one shortcut real engines take that this lesson glossed over. As written, the loop would re-run the whole growing sequence through the model on every trip. In practice, the work for earlier tokens is saved and reused, so each trip only processes the one new token. That saved work is the KV cache, and it depends on how attention works — lesson 6.

On a DGX Spark

Reproduce it: every number in this lesson is printed by lessons/05_sampling.py in the course’s companion code repository, llm-bytes-up-code. It needs the llama.cpp server described below; the repository’s scripts/start_server.sh starts one. The run this lesson quotes is recorded in that repository’s outputs/2026-10-12/, produced from a fresh clone on the Spark.

Everything here was run on the course’s DGX Spark on 2026-10-12, on the same model as lessons 1–4. The engine was llama.cpp’s llama-server, started temporarily and stopped afterwards:

llama-server -m deepseek-coder-6.7b-instruct.Q8_0.gguf \
    --host 127.0.0.1 --port 8099 -c 4096 -ngl 99 --parallel 1

The requests went to its /completion endpoint. These are the settings that matter for each experiment:

• Every probability in the settings table was re-derived by hand from the raw row: temperature by raising to the power 1/T and rescaling; top-k, top-p and min-p by cutting and rescaling.
  • Top-k, top-p, min-p and temperature 0.5 matched the server’s values exactly.
  • Temperature 1.5 matched to within 0.002. The by-hand version saw only the 200 most likely tokens, and flattening moves weight into the tail it could not see.
• The EOS token and the missing chat template were read from the GGUF metadata with gguf-py, as in lesson 1.