LLM Basics 3 — A Layer Is Matrix Multiplication

Series: 1 · What a model is on disk → 2 · Tokens and embeddings → 3 · A layer is matrix multiplication → 4 · The forward pass → 5 · Generating text → 6 · Attention and the KV cache → 7 · The hardware fit → Capstone · Why-LLM-Decode-Is-Memory-Bound

Lesson 2 ended with each token as a vector of 4,096 numbers. This lesson is about what one layer does to that vector. Almost all of it is one operation, repeated: multiply a matrix by a vector.

The specimen is still deepseek-coder-6.7b-instruct. Every number below was computed on its real weights, starting from the real embedding of " cat".

What is in a layer

Lesson 1 showed each of the 32 blocks holding nine tensors. A tensor is an array of numbers with any number of dimensions. In this model each one is either 1-D (a vector, like float v[4096]) or 2-D (a matrix, like float W[11008][4096]). Shapes are written rows × columns, in the same order as that C declaration. The nine are:

A block runs its two halves in sequence. In C-like pseudocode:

// x: float[4096], one token's vector on its way through the model
h = x + attention(rms_norm(x, attn_norm));   // half 1: look at other tokens
y = h + ffn(rms_norm(h, ffn_norm));          // half 2: transform this token
// y goes on to the next block

Attention is the only part of the model where one token’s vector is influenced by the other tokens; it is lesson 6. This lesson works through the feed-forward network (FFN), because it shows every ingredient of a layer with nothing else in the way.

Three tools first: dot product, length, and angle

Everything in this lesson is built from three small pieces of vector arithmetic. Here they are in two dimensions, where they can be checked by hand; the model does exactly the same thing with 4,096.

Dot product: multiply pairwise, then add

The dot product of two vectors of the same length multiplies matching elements and sums the results:

float dot(const float *a, const float *b, int n) {
    float s = 0;
    for (int i = 0; i < n; i++) s += a[i] * b[i];
    return s;
}

For a = [3, 4] and b = [4, 3]: 3×4 + 4×3 = 24.

The dot product is large when the two vectors have big values in the same places with the same signs, and small or negative when they don’t. But its size also depends on how large the vectors are, so on its own it mixes “how aligned” with “how big”.

Length: Pythagoras, in any number of dimensions

A vector’s length, also called its norm and written |a|, is the square root of the sum of its squared elements:

float length(const float *a, int n) { return sqrtf(dot(a, a, n)); }

For a = [3, 4]: √(9 + 16) = √25 = 5, the hypotenuse of a 3-4-5 triangle. The same formula works in 4,096 dimensions; you just can’t draw it.

Cosine similarity: the angle, with size divided out

To measure alignment alone, divide the dot product by both lengths:

cosine(a, b) = dot(a, b) / (|a| × |b|)

The result is the cosine of the angle between the two vectors:

For a = [3, 4] and b = [4, 3], both have length 5, so the cosine is 24 / (5 × 5) = 0.96, about 16°. They point nearly the same way.

The one operation: y = W · x

What a matvec is

Matvec is short for matrix-vector multiplication: taking a matrix and a vector and producing a new vector. Each output element is the dot product of one row of the matrix with the input vector.

A small example, a 2 × 3 matrix times a 3-element vector:

W = | 1  0  2 |      x = | 3 |
    | 0  1 -1 |          | 1 |
                         | 2 |

y[0] = row 0 · x = 1×3 + 0×1 + 2×2 = 7
y[1] = row 1 · x = 0×3 + 1×1 − 1×2 = −1

y = [7, −1]

Three inputs went in and two outputs came out, because the matrix has 3 columns and 2 rows.

As a loop

In general, a matrix W with R rows and C columns, multiplied by a vector x of length C, gives a vector y of length R:

for (int r = 0; r < R; r++) {
    float sum = 0;
    for (int c = 0; c < C; c++)
        sum += W[r][c] * x[c];
    y[r] = sum;
}

That loop is the whole of a matvec. Three things to see in it:

• Each output is a dot product of one row of W with the whole of x. The inner loop is the dot() function from above.
• The column count must match the input length, and the row count sets the output length. So ffn_gate, at 11008 × 4096, turns 4,096 numbers into 11,008.
• Every weight is used exactly once, for one multiply and one add.
  • A FLOP is one floating-point operation: one multiply, or one add.
  • So a matvec costs 2 FLOPs per weight.

Observed: the first output of ffn_gate · x for " cat" is −0.183733, whether computed by NumPy’s optimized routine or by the naive loop above. They are the same arithmetic.

What a row means

The dot product measures how closely two vectors line up, and lesson 2 used it, as cosine similarity, to find which tokens sit near each other. A matvec does that for every row at once:

• Each row of W is a learned pattern, a 4,096-number vector of its own.
• Each output says how strongly the input lines up with that row’s pattern.
• ffn_gate holds 11,008 such patterns. Applying it asks “how much does this token’s vector match each of them?” and gets 11,008 answers.

As with embeddings, no single row has a human-readable meaning. Training chose the patterns because they made predicting the next token easier.

Walking one token through the feed-forward network

The real numbers for " cat", through block 0’s FFN:

x            4096    the token's vector
  ↓ RMSNorm
xn           4096
  ↓ W_gate · xn        ↓ W_up · xn
g           11008      u      11008
  ↓ h = SiLU(g) × u    (element by element)
h           11008
  ↓ W_down · h
out          4096
  ↓ y = x + out        (residual)
y            4096    on to the next block

That is three matvecs, plus three cheap element-by-element steps between them. Taking the steps in order:

1. RMSNorm — rescale to a standard size

Root mean square (RMS) is a measure of how large a list of numbers typically is, ignoring sign. The name gives the recipe, read right to left:

1. Square each number. That makes every value positive.
2. Take the mean (the average) of the squares.
3. Take the square root of that mean, which undoes the squaring and returns to the original units.

Worked on [4, −2, 0, 4]:

Why not just average the numbers? The plain average of [4, −2, 0, 4] is 1.5, and the average of [5, −5] is 0 even though both numbers are large. Positive and negative values cancel. Squaring first stops that, so RMS measures size rather than balance.

RMSNorm divides every element by the RMS, which rescales the vector so its RMS becomes exactly 1. [4, −2, 0, 4] / 3 = [1.33, −0.67, 0, 1.33], whose RMS is 1. The pattern of the numbers is kept; only the overall scale changes. Then each element is multiplied by the matching entry of ffn_norm, a learned per-element gain:

float rms = sqrtf(mean(x[i] * x[i]) + eps);   // eps = 1e-6, read from the file
xn[i] = x[i] / rms * ffn_norm[i];

eps is a tiny constant that only matters when the vector is all zeros, where it prevents a division by zero.

• Observed for " cat":
  • Square: the first three elements are −0.1781, −0.0918, 0.1531, and their squares are 0.0317, 0.0084, 0.0234.
  • Mean: the mean of all 4,096 squares is 0.02917.
  • Root: √0.02917 = 0.1708, the RMS.
  • After dividing by it: the RMS is 1.0000.
• RMS and length are the same idea at different scales. Length is the square root of the sum of squares, and RMS is the square root of the mean. So length = RMS × √4096 = 0.1708 × 64 = 10.93, the length of " cat"’s vector used in step 5.
• Why it exists: across 32 blocks the vectors would otherwise drift to very large or very small magnitudes, and the arithmetic stops working well. Normalizing first keeps every block’s input on a standard scale.

2. Two matvecs up, 4,096 → 11,008

g = W_gate · xn and u = W_up · xn produce two 11,008-number vectors from the same input. The FFN works in a wider space — 11,008 is about 2.7× wider — and then comes back down.

3. SiLU gating — the one non-linear step

SiLU (Sigmoid Linear Unit) is a function from one number to one number. It is built from the sigmoid, a smooth dial that turns any number into a value between 0 and 1:

sigmoid(z) = 1 / (1 + e^(−z))          e ≈ 2.718, the base of natural logarithms
SiLU(z)    = z × sigmoid(z)

• For large positive z, e^(−z) is nearly 0, so the sigmoid is nearly 1 and SiLU(z) ≈ z. The value passes through.
• For large negative z, e^(−z) is huge, so the sigmoid is nearly 0 and SiLU(z) ≈ 0. The value is shut off.
• In between, it blends smoothly from one to the other.

For negative inputs SiLU never goes below about −0.28, so it can only pass a small value through.

The FFN uses it as a gate: h[i] = SiLU(g[i]) × u[i], element by element, for all 11,008 positions. g decides how much of u gets through, so g acts as 11,008 valves on u.

• Observed for " cat":
  • 76% of the gate values g are negative. Each of those scales its u value by a factor between −0.28 and 0.
  • 91% of the valves have |SiLU(g)| < 0.1, so they pass less than a tenth of their u value.
  • 6% have |SiLU(g)| < 0.01, so they are effectively closed.
  • A minority of positions pass their u through at full strength or more.
• This gate-times-value design is called SwiGLU, and nearly every current LLM uses it.

This step is the reason a layer can do anything matrices alone cannot. That depends on what “linear” means.

• A function is linear when scaling the input scales the output by the same amount, and adding inputs adds outputs.
• A matvec is linear. W · (2x) = 2 × (W · x).
• SiLU is not. SiLU(2) = 1.76, but 2 × SiLU(1) = 1.46.

Linear functions have a property that matters here: two linear steps in a row are always equivalent to one. W_down · (W_up · x) equals (W_down · W_up) · x, and that product is just another matrix. Without something non-linear between the matrices, all 32 layers would collapse into a single matrix multiply, no more capable than one layer. The non-linearity is what makes depth mean anything.

4. One matvec down, 11,008 → 4,096

out = W_down · h returns to the model’s working width.

5. Residual — add, don’t replace

y = x + out. The block adds an adjustment to the token’s vector rather than replacing it.

• Observed: the length of " cat"’s vector is 10.93, and the adjustment’s length is 3.70, about a third as long.
• How far did the direction change? That is the cosine similarity from the start of this lesson, between the vector before (x) and after (y). The table below works it out.

A cosine of 0.95 is an angle of about 18°. So the vector was nudged, not overwritten: it still points mostly where it did, and now carries the block’s adjustment.

• Why it exists: each block only has to learn a correction, and information from the embedding survives all 32 blocks. That stream of vectors, adjusted block after block, is called the residual stream.

Honesty note: in the real model this FFN receives the output of block 0’s attention half, not the raw embedding. Feeding it the raw embedding here keeps the demo to one half of one block. The weights and arithmetic are real, but the intermediate values are not exactly what the model computes in a full run.

Why “a layer is matrix multiplication”

Counting the work for one token through one whole block (computed):

The cheap steps are what make a layer work; the matvecs are what make it cost. That is why AI hardware is built around matrix-multiply engines — NVIDIA’s tensor cores are hardware units that do nothing else, each computing a small matrix tile (such as 16 × 16) in a single operation — and why “how fast does it run” is essentially “how fast can it multiply matrices, and feed them.”

The attention half (lesson 6) adds one more ingredient, comparing a token against the earlier tokens. Its four big projections, attn_q, attn_k, attn_v, and attn_output, are themselves four ordinary 4096 → 4096 matvecs.

One token versus many: matvec versus matmul

With one token, x is a single vector, and each weight is used once: 2 FLOPs for every weight read from memory.

With N tokens processed together — a prompt, or a batch of users — x becomes a 4096 × N matrix. The same loop runs N times per weight while the weight is already loaded:

for (int r = 0; r < R; r++)
    for (int c = 0; c < C; c++) {
        float w = W[r][c];                 // read once
        for (int n = 0; n < N; n++)        // used N times
            Y[r][n] += w * X[c][n];
    }

That is matrix-matrix multiplication (“matmul”, or GEMM), and it gets N times the work out of each weight read. The difference is measurable even on a CPU.

Observed on the DGX Spark’s CPU (NumPy, ffn_gate dequantized to FP32, 180 MB):

256 times the work took 20 times as long. The single-token case is paced by how fast 180 MB can be read: 128 GB/s is about half the machine’s 273 GB/s peak, with the arithmetic units mostly waiting. That is the whole of the capstone’s argument in one line of measurements, and lesson 7 picks it up for the GPU.

On a DGX Spark

Reproduce it: every number in this lesson is printed by lessons/03_ffn_by_hand.py in the course’s companion code repository, llm-bytes-up-code. The run this lesson quotes is recorded in that repository’s outputs/2026-10-12/, produced from a fresh clone on the Spark.

Everything here was run on the course’s DGX Spark on 2026-10-12, against the same GGUF file as lessons 1 and 2.

• The weights: blk.0.ffn_gate, ffn_up, ffn_down, and ffn_norm were read with llama.cpp’s gguf-py and dequantized from Q8_0 to FP32 by hand, using lesson 1’s block layout.
• The model’s dimensions came from its metadata, not from assumption:
  • llama.embedding_length = 4096
  • llama.feed_forward_length = 11008
  • llama.block_count = 32
  • llama.attention.layer_norm_rms_epsilon = 1e-6
• The FFN was computed in NumPy exactly as written above, with one output element re-checked with a plain Python loop.
• The timing ran on the Spark’s Arm CPU, not its GPU.
  • Each figure is the mean of repeated runs after warm-up, on an otherwise idle machine (load average 0.01).
  • The 256-token input was random data, because only the timing mattered.
  • A GPU’s numbers are far higher, but the ratio between the two rows is the point, and it holds on any processor.