LLM Basics 4 — The Forward Pass

Series: 1 · What a model is on disk → 2 · Tokens and embeddings → 3 · A layer is matrix multiplication → 4 · The forward pass → 5 · Generating text → 6 · Attention and the KV cache → 7 · The hardware fit → Capstone · Why-LLM-Decode-Is-Memory-Bound

Lessons 1–3 took the model apart. This lesson puts it back together and runs it.

A forward pass is one complete run of the model: input at the front, every layer in order, output at the back. “Forward” is the direction the data flows. (Training also has a backward pass, which adjusts the weights. This course does not cover training.)

What goes in: a sequence of tokens. What comes out: one probability for every one of the 32,256 entries in the vocabulary — the model’s estimate of what the next token will be. That is the model’s entire output. It does not produce words or answers; it produces a probability for every possible next token.

The specimen is still deepseek-coder-6.7b-instruct, now run for real on the DGX Spark’s GPU.

The whole model in fifteen lines

// tokens: int[N], the prompt after tokenizing (lesson 2)
float X[N][4096];

for (int n = 0; n < N; n++)                     // 1. embed: one row lookup per token
    X[n] = token_embd[tokens[n]];               //    (lesson 2)

for (int b = 0; b < 32; b++)                    // 2. 32 blocks, in order (lesson 3)
    X = block(b, X);                            //    attention, then feed-forward

float *last = X[N - 1];                         // 3. keep only the last position
last = rms_norm(last, output_norm);             // 4. one final normalization
float logits[32256] = matvec(output, last);     // 5. one score per vocabulary entry
float probs[32256]  = softmax(logits);          // 6. scores -> probabilities

Each line is something earlier lessons covered, plus two new ideas — logits and softmax — explained below. A brief reminder of the terms:

• Embedding (lesson 2): each token ID selects one 4,096-number row of token_embd.
• Matvec (lesson 3): matrix-vector multiplication. Each output number is the dot product — multiply pairwise, then add — of one matrix row with the input vector.
• RMSNorm (lesson 3): rescale a vector so its typical element size (its root mean square) is 1.

All positions travel together

A prompt of N tokens enters as N vectors — an N-row array — and all N go through every block together.

• The feed-forward half treats each position on its own. It runs the same three matvecs on every row.
• Attention is the only place positions interact. It lets each position take in information from the positions before it. That is how the vector at the end of “The capital of France is” comes to carry information about “France”. Lesson 6 covers how.
• After 32 blocks, every position holds a vector, but predicting the next token only needs the last one. It is the only position that has seen the whole prompt. Inference engines typically compute the final step (lines 4–6) for the last position alone.

The exit: logits

output.weight is a matrix of 32,256 rows × 4,096 columns — one row per vocabulary entry. Multiplying it by the final vector is one more matvec, and it produces 32,256 numbers.

Lesson 3 described each row of a matrix as a learned pattern, with each output measuring how well the input matches it. Here each row is the pattern for one token, so each output scores how well the final vector matches that token.

Those 32,256 raw scores are called logits.

• A logit can be any number: positive, negative, large or small.
• Higher means more likely. But a logit is not yet a probability: logits don’t fall between 0 and 1, and they don’t add up to anything in particular.

Softmax: turning scores into probabilities

Softmax converts a list of logits into probabilities — numbers between 0 and 1 that add up to exactly 1. The recipe:

1. Raise e (≈ 2.718, the base of natural logarithms) to the power of each logit. Every result is positive.
2. Add those results up.
3. Divide each one by the total.

Worked on three logits, [2.0, 1.0, 0.1]:

Why this particular recipe:

• Exponentiating makes everything positive while keeping the order: the highest logit still wins.
• Only the gaps between logits matter. Adding the same amount to every logit changes nothing, because it scales every e^logit by the same factor, and that factor cancels in the division.
• Each gap of 1 multiplies the odds by e ≈ 2.7. Token A’s logit is 1 higher than B’s, so A is 2.7× as likely. A gap of 5 is a factor of about 150, so a clear favorite takes nearly all the probability.

Why it is called softmax. A plain max — “argmax” in code — would give the top-scoring token 100% and every other token 0%. Softmax is the soft version: the highest score still gets the largest share, but every runner-up keeps some probability. That leftover share is what lets the sampling settings in lesson 5 sometimes pick a token other than the favorite.

Log-probabilities

Tools usually report a logprob instead of a probability. It is the natural logarithm of the probability, ln(p), so p = e^logprob.

• A logprob is always ≤ 0, and 0 means certainty.
• Logprobs are used because the probabilities of rare tokens get extremely small, and logs keep them manageable.
• Example: llama.cpp reported a logprob of −0.0756 for " Paris" below, and e^−0.0756 = 0.927, or 92.7%.

Real outputs

Four prompts, each run through a full forward pass on the real model, showing the eight most probable next tokens. All values are observed.

“The capital of France is”

“The cat sat on the”

def fibonacci(n):⏎ if n

for (int i = 0; i <

What these show:

• Familiar text gives a confident distribution. " mat" at 98.5% and " Paris" at 92.7%.
• Genuinely open choices give a spread-out one. if n == 0, if n <= 1 and if n < 2 are all correct ways to start a Fibonacci function, and the model splits its probability across them: its top two are 31% and 28%. The model does not choose — it reports the spread. Choosing is lesson 5.
• The loop bound is the most uncertain of all. The top eight tokens cover only 58% of the probability, because any variable name could follow i <.
• The model’s training shows through. This is a code model, so even after “The capital of France is” its runners-up are code punctuation — ", <, { — rather than words.
• Tokens include their spacing. == with spaces on both sides is one token, and == without spaces is a different one. The model assigns each its own probability (lesson 2).

What one forward pass costs

• Memory: every weight is read once — essentially the whole 7 GB file (lesson 1).
• Arithmetic: about 2 FLOPs per parameter per token. A FLOP is one floating-point operation, a multiply or an add. So the cost is 2 × 6.74 billion ≈ 13.5 billion FLOPs per token, almost all of it matvecs (lesson 3).

Observed on the Spark’s GPU:

The prompt was processed about 72 times faster per token than new text was generated. These two phases have names:

• Prefill is processing the prompt. All its tokens are already known, so they go through the model together, and each weight is read once for all of them.
• Decode is generating new tokens. Each new token depends on the one before it, so each needs a forward pass of its own, and each pass reads every weight again.

That difference — the same weights, read once for 1,921 tokens versus once per token — is the matvec-versus-matmul effect measured in lesson 3, now on the whole model. Two more details foreshadow later lessons:

• 33.2 tokens/s is about 85% of the 39 tokens/s ceiling computed in lesson 1 from the Spark’s memory bandwidth. Decode speed is set by how fast the weights can be read (the capstone).
• Decode slowed from 33.2 to 29.6 tokens/s after the long prompt. The weights didn’t change, but attention has more earlier positions to read (lesson 6).

Where this leaves us

One forward pass produces one probability distribution over the next token. It does not produce text. Getting text means picking a token from that distribution, appending it, and running another forward pass — over and over. That loop, and the choices hidden in “picking”, is lesson 5.

On a DGX Spark

Reproduce it: every number in this lesson is printed by lessons/04_next_token.py in the course’s companion code repository, llm-bytes-up-code. It needs the llama.cpp server described below; the repository’s scripts/start_server.sh starts one. The run this lesson quotes is recorded in that repository’s outputs/2026-10-12/, produced from a fresh clone on the Spark.

Everything here was run on the course’s DGX Spark on 2026-10-12.

• The engine was llama.cpp’s llama-server, started temporarily with this model and stopped afterwards:

llama-server -m deepseek-coder-6.7b-instruct.Q8_0.gguf \
    --host 127.0.0.1 --port 8099 -c 2048 -ngl 99 --parallel 1

• -ngl 99 puts every layer on the GPU.

• -c 2048 sets a 2,048-token context window.

• --parallel 1 serves one request at a time.

• The probabilities came from its /completion endpoint:

curl -s 127.0.0.1:8099/completion -H 'Content-Type: application/json' -d '{
  "prompt": "The capital of France is",
  "n_predict": 1, "n_probs": 8, "temperature": 0,
  "post_sampling_probs": false, "cache_prompt": false }'

• n_probs asks for the top 8 tokens.

• post_sampling_probs: false returns probabilities from the full softmax, before any of the adjustments lesson 5 covers.

• temperature: 0 makes the server pick the single most likely token; lesson 5 explains temperature.

• cache_prompt: false forces every prompt to be processed from scratch.

• The speeds are the server’s own timings report. The 1,921-token prefill ran twice, at 2,379 and 2,408 tokens/s.

  • The Spark’s two resident model servers stayed loaded throughout, but were idle (load average 0.02).
  • The decode rate is single-request speed.

• The same server cross-checked lesson 2. Its /tokenize endpoint corrected one tokenization in that lesson’s table.