LLM Basics 1 — What a Model Is on Disk

Series: 1 · What a model is on disk → 2 · Tokens and embeddings → 3 · A layer is matrix multiplication → 4 · The forward pass → 5 · Generating text → 6 · Attention and the KV cache → 7 · The hardware fit → Capstone · Why-LLM-Decode-Is-Memory-Bound

Before any of the interesting questions about LLMs — how they “think”, why they are slow, why they need GPUs — it helps to know what one physically is. A model is a file of numbers. Everything in this note comes from opening a real one.

The specimen is deepseek-coder-6.7b-instruct.Q8_0.gguf: a 7,163,879,648-byte file holding a plain dense Llama-architecture model. It is old and unremarkable, which makes it a clean example. Values marked observed were read from the file; values marked computed are arithmetic on those.

A model is two things, and the file holds only one

• The architecture is code.
  • A fixed program: for each layer — multiply by this matrix, then that one, normalize, repeat.
  • It lives in the inference engine (llama.cpp, vLLM, TensorRT-LLM), not in the model file.
  • The file only names it. Observed: general.architecture = llama. The engine reads that string and runs its built-in Llama routine.
• The weights are data.
  • Billions of numbers, and no instructions anywhere among them.
  • “Downloading a model” means downloading this data file.

In C terms: a program that declares a huge const float W[] it does not contain, and mmaps it from a file at startup. Give it a different file with the same architecture and shapes and the same program runs a different model.

The file is laid out like an ELF binary

First, one word the rest of the course uses constantly, tensor. A tensor is an array of numbers, all of one type, with any number of dimensions. In machine learning that is all it means.

• A tensor’s shape is the list of its dimension sizes — [11008, 4096] for that matrix.
• This course writes a matrix’s shape as rows × columns, in the same order as a C declaration: float W[11008][4096] is an 11008 × 4096 matrix — 11,008 rows, each 4,096 numbers long.
• Its dtype is the type of every element in it.
• The word is borrowed from mathematics and physics, where it means something more specific. In ML code it is simply the general name for an array of any dimension, which is why frameworks call their array type Tensor.

In this model every tensor is 1-D or 2-D, so read “tensor” as “vector or matrix” throughout.

With that, the file layout:

Other formats (safetensors, PyTorch .bin) differ in detail but have the same three parts.

Observed: the file holds 291 tensors. The first block of them:

The names carry the whole structure:

• blk.N is a layer, or “block”. There are 32, numbered 0–31, all the same shape.
  • 32 × 9 tensors plus 3 at the ends gives the 291.
  • Each block has four attention matrices (attn_*) and three feed-forward matrices (ffn_*). What they do is step 3.
• token_embd is at the front and output is at the back.
  • Both have 32,256 rows: one per entry in the model’s vocabulary.
  • They are the doorways between text and numbers — step 2.
• 4096 is the model’s working width, and it appears in nearly every shape.

GGUF prints shapes fastest-varying dimension first, so the file’s own listing shows token_embd as [4096, 32256]. Read it as 32,256 rows of 4,096 contiguous numbers — one row per vocabulary entry.

“6.7B parameters” is a literal count

A parameter is one number in one of those arrays. Adding up the table:

• One block: 4 × 16.78M + 3 × 45.09M + 2 × 4,096 = 202.4M
• 32 blocks: 6,476.2M
• Plus embedding, output, and final norm: 6,740,512,768

That is computed, and it equals the file’s own total exactly. It is where the “6.7B” in the name comes from. For a “27B” model the same sum comes to about 27 billion.

What the bytes look like

How many bytes each parameter takes is its dtype, chosen when the file is produced:

Q8_0 is a block-quantization scheme. Weights are stored in groups of 32: one 2-byte FP16 scale, followed by 32 signed 8-bit integers. Each real weight is scale × int8. That makes 34 bytes per 32 weights.

The first 34 bytes of blk.0.attn_q.weight (observed):

63 11 | 00 81 08 07 f4 0d fb fb fd f5 09 fd f5 00 fc f9 01 05 fb 01 16 0c 03 fe 09 f1 f8 fd 0f 11 fc f0
scale | 32 x int8

Decoded (computed):

• Scale: 0x1163 as little-endian FP16 ≈ 0.000658.
• Integers: 00 81 08 07 f4 → 0, −127, 8, 7, −12.
• So layer 0’s Q matrix begins: 0.0, −0.0835, 0.0053, 0.0046, −0.0079, …

That is a model: about 6.7 billion small numbers like these. None of it is readable. No region holds “Python syntax” and no byte holds a fact. Training set every value by nudging all of them a tiny amount at once over trillions of examples, so whatever the model “knows” is spread across all of them.

Why this matters later

To produce one token, the engine reads every blk.* matrix and output.weight once — essentially the whole file. That single fact is why LLM decoding speed is set by memory bandwidth, worked through in Why-LLM-Decode-Is-Memory-Bound.

The one exception is token_embd. It is read like an array index, one 4,096-number row per token, which is the subject of LLM-Basics-2-Tokens-and-Embeddings.

On a DGX Spark

Reproduce it: every number in this lesson is printed by lessons/01_inspect_gguf.py in the course’s companion code repository, llm-bytes-up-code. The run this lesson quotes is recorded in that repository’s outputs/2026-10-12/, produced from a fresh clone on the Spark.

The lab machine for this course is an NVIDIA DGX Spark (GB10 superchip): 128 GB of LPDDR5X memory shared by the CPU and GPU, at 273 GB/s.

• Where the specimen came from: it ships with NVIDIA’s DGX Spark playbooks, as one of the multi-agent-chatbot assets. It was read on the Spark on 2026-10-12 with the gguf Python reader from llama.cpp’s gguf-py directory.
• The key calls, if you want to explore on your own: open the file with gguf.GGUFReader, and iterate reader.tensors for name, shape, type, element count, and byte offset. The metadata is in reader.fields.
• What this model’s size means on the Spark: about 7 GB is read per generated token, so the speed ceiling is 273 GB/s ÷ ~7 GB ≈ 39 tokens/s.
  • That is computed from the spec bandwidth, not measured.
  • Real throughput usually lands at roughly half to three-quarters of a ceiling computed this way.
• Measured on 2026-10-12: run on the Spark’s GPU with llama.cpp, this model generated 33.2 tokens/s — about 85% of that 39 tokens/s ceiling. Lesson 4 has the details.