Why LLM Decode Is Memory Bound

Series: 1 · What a model is on disk → 2 · Tokens and embeddings → 3 · A layer is matrix multiplication → 4 · The forward pass → 5 · Generating text → 6 · Attention and the KV cache → 7 · The hardware fit → Capstone · Why decode is memory bound

This course started with one sentence from an article explaining why speculative decoding is fast (jbarrow.ai, 2026-10-08). The article calls single-token decoding “very memory bound”, and says:

“For every token we have to stream all the active parameters of the model.”

At the start that sentence raised two questions: stream them from where to where, and why does that set the speed? Seven lessons later, both have measured answers. This capstone puts them together, then uses them to explain the four standard ways to make generation faster.

Two terms carry the whole argument:

• Decode is the phase where a large language model (LLM) generates its answer, one token at a time (lesson 5). Each new token needs a complete run through the model — a forward pass (lesson 4) — and can’t start until the token before it has been chosen.
• Prefill is the phase before that, where the model reads the prompt. All of the prompt’s tokens are already known, so they go through the model together.

Everything below uses the course’s model, deepseek-coder-6.7b-instruct at 8-bit (Q8_0), on the course’s DGX Spark, unless it says otherwise. Figures marked observed were measured; figures marked computed are arithmetic.

Stream from where to where

• From main memory. The model’s weights live in the GPU’s main memory: about 7 GB for the course’s model (lesson 1).
  • On a discrete GPU such as an H100, that is HBM (high-bandwidth memory) on the GPU’s own board — the “VRAM” on the spec sheet.
  • On the DGX Spark, it is 128 GB of LPDDR5X shared by the CPU and GPU.
• To the arithmetic units. The GPU’s streaming multiprocessors (SMs) can only compute on values held in their registers. Their tensor cores — hardware that multiplies small matrix tiles in a single operation — are where nearly all of an LLM’s arithmetic happens (lesson 7).
• Through caches far too small to help. The largest cache on the Spark’s GPU, the L2, is 24 MB (observed). The model is about 300 times larger.

So the weights cannot stay near the arithmetic. They are read in, used once, and pushed out to make room for the next ones. “Streaming the parameters” means copying every weight from main memory into the SMs — and every generated token does it again.

Why “the weights don’t move” is only half right

A common explanation says the weights are fixed pipes and each token flows through them. That is true about residency: the weights are loaded into memory once, at startup, and never change while the model runs.

It is false about traffic. Each weight is copied into the compute units every time a token needs it, because that is the only place it can be used. The weights stay in memory, and every token reads all of them again.

What one generated token costs

Lesson 3 showed that 99.97% of an LLM’s work is matrix-vector multiplication (“matvec”): each output is the dot product of one matrix row with the input vector, so every weight is used exactly once, for one multiply and one add. For one token, a forward pass is in effect:

// one generated token: a full pass over ~7 GB of weights that no cache can hold
for (size_t i = 0; i < N_WEIGHTS; i++)
    acc[row(i)] += W[i] * x[col(i)];   // read a weight, use it once, move on

That has the shape of a giant memcpy with one multiply-add per element. It is limited by how fast memory can deliver bytes, not by how fast the processor can do arithmetic. That is what memory bound means: the time is set by bytes moved, not operations performed.

The numbers for one token:

• Reading takes about 180 times longer than computing. The ceiling this sets is 1 token per ~27 ms, or about 37 tokens/s.
• Observed: llama.cpp generated 32.9–33.7 tokens/s across the course’s runs — about 90% of that ceiling (lessons 4 and 7, and the batching run below).
• During those ~30 ms per token, the tensor cores are busy for about 0.15 ms, roughly 0.5% of the time. The rest of the time they are waiting for weights.

The article’s own example scales the same way. Qwen3.5-27B stored at 16 bits is 27 billion × 2 bytes ≈ 54 GB (computed).

Tokens per step

A step is one forward pass, and each weight is read from memory once per step. In decode, a step normally produces one token for one user. But nothing about a weight ties it to one token. Once W[i] is in a register, it can be applied to several tokens’ vectors before it is discarded:

for (size_t i = 0; i < N_WEIGHTS; i++) {
    float w = W[i];                     // one read from memory
    for (int t = 0; t < T; t++)         // T tokens share it
        acc[t][row(i)] += w * x[t][col(i)];
}

T is tokens per step. The memory traffic is the same; the useful work is T times larger. Lesson 3 called this the difference between a matvec (one token) and a matmul (many). There are three ways to get T above 1:

• Prefill. The prompt’s tokens are all known, so they share every read.
  • Observed (lesson 4): a 1,921-token prompt was read at ~2,400 tokens/s, against 33 tokens/s for generation.
• Batching: serving several users at once, with one token from each user’s answer in every step.
• Speculative decoding: guessing several of one user’s future tokens and checking them in a single step (below).

How far “free” goes: the crossover

Extra tokens in a step are nearly free only while reading the weights is still the slower of the two costs. Lesson 7 measured where that stops, by multiplying one real weight matrix (11008 rows × 4096 columns, 16-bit) by T token vectors at once. Observed:

The crossover can be predicted:

• A job’s arithmetic intensity is its FLOPs per byte read.
  • One token through 16-bit weights does 2 FLOPs per 2-byte weight: 1 FLOP per byte.
  • T tokens sharing each read do T FLOPs per byte.
• A machine’s ridge point is its peak compute divided by its memory bandwidth — the intensity at which both run out together. (The name comes from the roofline chart engineers use to plot this.)

So on either machine, a step stays memory bound until roughly 300 tokens share each weight read. The Spark’s knee was measured between 256 and 512. Single-user decode runs at T = 1, about three hundred times below the point where the GPU’s arithmetic would be the limit.

What sets the ridge is the ratio, not either number alone. The Spark has about a twelfth of an H100’s bandwidth, and proportionally less compute too, so its ridge lands in the same place. More bandwidth for the same compute would lower the ridge; less would raise it.

Batching, measured

Observed on the Spark: one llama.cpp server, with 1 to 16 users each generating 256 tokens at the same moment.

• Sixteen users got almost ten times the total output, while each user’s speed fell by about a third.
• The server paid for each weight read once and spent it on up to 16 tokens.
• Why each user still slowed down: each user brings their own KV cache — the saved attention work for their conversation, which grows with every token (lesson 6).
  • That cache is not shared, so 16 users mean 16 caches to read on every step.
  • Each user’s attention computation is also their own.
  • That is an explanation consistent with lesson 6, not something this run isolates.
• This is why a busy server is efficient and a lone user is not. A provider serving many users at once runs the same hardware much closer to its arithmetic limit than one person chatting with a local model ever does.

Why speculative decoding is fast

Batching helps a server, but a single user still generates at T = 1. Speculative decoding raises T for one user:

1. A draft model guesses ahead. A much smaller, cheaper model, whose own weight reads cost little, guesses the next K tokens — say 5.
2. The real model checks all of them in one step. It runs a single forward pass over the 5 guessed tokens: one read of the weights instead of five.
3. It keeps the guesses it agrees with. That pass gives the real model’s own prediction at every position. It accepts guesses up to the first one that differs from what it would have chosen, then adds its own token at that point.

Because T = 5 is far below the ridge point, the checking step costs almost exactly what a T = 1 step costs. If 4 of the 5 guesses are accepted, one weight read yields 5 tokens. Speculative decoding turns idle arithmetic into tokens, and it works only because decode is memory bound.

The output is the same as the real model would produce on its own. The draft only proposes; the real model decides every token.

Observed on the same Spark, on 2026-08-25, with a larger model — Qwen3.6-35B-A3B at 4-bit, served by llama.cpp:

• MTP (multi-token prediction) means the model carries its own small draft head, so no separate draft model is needed.
• The gain is larger on factual answers because predictable text means more guesses are accepted.

The same lever, four ways

Every standard way of speeding up decode reduces bytes read per token produced:

The last row explains the article’s word “active”.

• A MoE model’s active parameters are the ones a token actually uses. That is the routed experts plus the parts every token shares.
• A 235B-total, 22B-active model reads about 22 billion parameters per token, not 235 billion. That is why a large MoE model can generate faster than a much smaller dense one — a model that uses all its weights for every token.

On the Spark (computed, from its 273 GB/s specified bandwidth ÷ bytes read per token):

• Quantization: going from 8-bit to 4-bit roughly doubles the dense model’s ceiling.
• MoE: a 120B MoE model has a higher ceiling than a 27B dense one, because it reads less per token.
• Measured against these: only the last row has been checked against a real run, at the 62 tokens/s above — about half its ceiling. Use roughly half the computed ceiling as a working estimate, and measure before quoting a number to a customer.

What this leaves out

• The KV cache adds bytes too. Long conversations and large batches add cache reads to every step, on top of the weights (lesson 6).
• Prefill and decode are different workloads. Prefill is compute bound and decode is memory bound. That is why large serving systems increasingly run them on separate GPUs, sized for each — disaggregated serving, as in NVIDIA Dynamo.
• Real software reaches less than the peak. The course’s decode ran at about 90% of its bandwidth ceiling, and its prefill at well under peak compute (lesson 7). The ceilings bound what is possible; they do not predict it exactly.

The sentence, again

“For every token we have to stream all the active parameters of the model.”

With the course behind it, the sentence says:

• For every token: in decode, each generated token is its own forward pass, because it depends on the token before it.
• Stream: copy from the GPU’s main memory into its compute units, because the caches are hundreds of times too small to keep the model nearby.
• All the active parameters: every weight the token uses — the whole model if it is dense, or only the routed experts if it is a mixture of experts.
• Very memory bound: the arithmetic takes a fraction of a percent of the time; the reading takes the rest. On the course’s model and machine, that is ~0.15 ms of arithmetic inside ~30 ms of reading.

Every technique in the article — and every technique in this note — is a way of getting more tokens out of each byte read.

On a DGX Spark

Reproduce it: the measurements in this note come from the course’s companion code repository, llm-bytes-up-code:

The runs are recorded in that repository’s outputs/2026-10-12/, produced from a fresh clone on the Spark.

• The batching run used llama.cpp build 10630 on the Spark’s GPU, with greedy decoding, 256 tokens per user, and a different prompt for each user.
  • The Spark’s two resident model servers stayed loaded but idle.
• The speculative-decoding measurements come from the Spark’s own serving setup on 2026-08-25, not from the companion repository. They used a different, larger model, and are kept here because they are real measurements of the mechanism on the same hardware.
• The H100 figures are from NVIDIA’s specifications (dense 16-bit throughput and HBM3 bandwidth), not measured.