LLM Basics 2 — Tokens and Embeddings

Series: 1 · What a model is on disk → 2 · Tokens and embeddings → 3 · A layer is matrix multiplication → 4 · The forward pass → 5 · Generating text → 6 · Attention and the KV cache → 7 · The hardware fit → Capstone · Why-LLM-Decode-Is-Memory-Bound

A model only does arithmetic, so text has to become numbers before anything else can happen. That takes two steps, and neither involves any math:

1. Text → token IDs. The tokenizer splits text into pieces from a fixed vocabulary and replaces each piece with its index. Integers.
2. Token ID → vector. Each ID selects one row of the embedding table, token_embd. That gives 4,096 numbers per token.

The examples use the same specimen as step 1, deepseek-coder-6.7b-instruct. Everything below was observed by running its vocabulary and embedding table directly.

First: text is already numbers

Before a tokenizer gets involved, text on disk or in memory is already a sequence of numbers: bytes.

A byte is a number from 0 to 255

• A byte is 8 bits. Eight binary digits give 2⁸ = 256 possible patterns, from 00000000 to 11111111.
  • Read as an unsigned integer that is 0–255, written in hex as 00–FF.
  • In C it is unsigned char.
• “The 256 possible bytes” just means every value one byte can hold. No byte value exists outside that list, so any file of any kind — text, image, executable — is a sequence drawn from those 256 values.

Which number stands for which character

That mapping is a character encoding, and for text today it is almost always UTF-8.

• Bytes 0–127 are ASCII, unchanged since the 1960s, at one byte per character.
  • Letters, digits and punctuation: A = 65 (0x41), a = 97 (0x61), 0 = 48 (0x30).
  • Space = 32 (0x20).
  • Control codes that are not visible characters: tab = 9 (0x09), newline = 10 (0x0A).
• Everything else takes two to four bytes, each in the range 128–255.
  • Accented letters, other alphabets, CJK characters and emoji are built this way.
  • Observed with Python’s str.encode("utf-8"):

So “the cat” in a text file is literally the 7 bytes 74 68 65 20 63 61 74. For the full story of encodings, see Text Encoding from First Principles.

Why this matters for tokenizing: if the vocabulary contains all 256 single-byte values, then any input whatsoever can be represented. In the worst case it is spelled one byte at a time. That is the safety net under everything below.

Step 1: text → token IDs

The vocabulary is a string table

The GGUF metadata holds the vocabulary as an array of 32,256 strings (tokenizer.ggml.tokens). A token ID is just an index into that array. It is an enum with 32,256 members.

const char *vocab[32256] = { "!", "\"", "#", "$", ... };   // ID = position

Some entries, by ID:

The Ġ and Ċ characters are bytes in disguise

The vocabulary is stored as strings, and some bytes are awkward to store or display that way: space, tab, newline, and the 128–255 range that makes no sense on its own. So this tokenizer family (the “byte-level BPE” first used by GPT-2) gives each of the 256 bytes a visible stand-in character:

• Printable ASCII bytes stand for themselves. A is A.
• The rest are shifted to visible characters from further up Unicode.
  • Observed:

So Ġcat means " cat", with the leading space, and ĉint is a tab followed by int. The odd characters are purely a display convention; underneath, every token is a string of bytes.

Observed: all 256 byte stand-ins are present in this vocabulary, with none missing. That is the safety net from the previous section, checked rather than assumed.

The space is part of the token. " cat" (5939) and “cat” are different IDs, so the model sees “cat” mid-sentence and at the start of a word as different inputs.

Real tokenizations

Patterns worth noticing:

• Common words are one token each — " cat", " the", " return", " configured".
• Rare words are split into pieces. “antidisestablishmentarianism” is 8 tokens.
• Names and acronyms fragment badly. “DGX” is three tokens; “Spark” is two.
• Code has tokens for its own idioms: ): is one token, and four spaces of indentation become ĠĠĠ plus the space glued to return.
• Every tokenization round-trips exactly. Decoding the IDs reproduces the original bytes.
• ␣ marks a leading space in the Text column. configured mid-sentence is one token, but at the very start of a text, with no space before it, configured splits into configure + d — another case of the space being part of the token.

Why pieces, and not characters or words

How the vocabulary was built: BPE

Byte Pair Encoding is a dictionary compressor, close in spirit to LZW:

1. Start with the 256 possible byte values as the vocabulary.
2. In a large training corpus, find the most frequent adjacent pair — say t + h.
3. Add th as a new entry, and record the merge.
4. Repeat until the vocabulary reaches its target size.

Observed: this file stores 31,757 recorded merges (tokenizer.ggml.merges). Encoding text replays those merges in the order they were learned, which is deterministic — the tokenizer is a lookup procedure, not a neural network.

Because the 256 single bytes are always in the vocabulary, any input can be encoded: emoji, binary garbage, a language the corpus barely saw. It just costs more tokens. Observed in this tokenizer:

All three decode back to the exact original text. Rare text is never rejected — it just gets more expensive.

Rule of thumb, and why counts matter

• English prose: roughly 4 characters per token. Code, names, numbers, and most non-English text tokenize worse.
• Everything LLM is priced and measured in tokens: context windows, API cost, and tok/s.
• Different models have different tokenizers. The same text is a different number of tokens on Llama, Qwen, and GPT models, so token counts don’t transfer between them.

Step 2: token ID → vector

The embedding table is a 2-D array, and lookup is indexing

token_embd.weight has one row of 4,096 numbers per vocabulary entry: 32,256 rows × 4,096 columns. Converting a token to a vector is an array index — no multiplication:

float E[32256][4096];          // token_embd, loaded from the file
float *x = E[token_id];        // the token's vector: 4096 floats

Observed: row 5939, " cat", begins

-0.1781  -0.0918  +0.1531  +0.2644  -0.1252  -0.1837  -0.0891  +0.0111  ... (4096 values)

For a sentence of N tokens, the model now holds an N × 4,096 array. That array is what flows through the 32 layers. The token IDs are never used again on the way in.

Why a vector of 4,096 numbers?

An ID is just a label. ID 5939 is not “closer” to 5940 than to 12, so the integer itself tells the model nothing. A vector is a position in a 4,096-dimensional space, and training moves tokens that are used in similar ways toward each other.

The standard closeness measure is cosine similarity, which is built from two simpler pieces:

• Dot product: multiply the two vectors element by element, then add up the results. It comes out large when both vectors have big values in the same places with the same signs.
• Length: the square root of the sum of a vector’s squared elements — Pythagoras, in 4,096 dimensions. For [3, 4] that is √(9 + 16) = 5.

Cosine similarity = dot product ÷ (length of one × length of the other). Dividing by the lengths removes size, leaving only direction. The result is the cosine of the angle between the two vectors:

• +1: same direction (0°)
• 0: unrelated (90°)
• −1: opposite directions (180°)

Lesson 3 works an example through by hand.

Observed similarities between rows of this model’s embedding table:

Observed nearest neighbours across all 32,256 rows:

• Monday → onday, Friday, Saturday, Sunday, Tuesday, Wednesday, uesday, Thursday
• int → ⇥int, int, Int, Int, float, uint, bool, INT
• def → def, Def, Def, DEF, fun, ef, void, function
• cat → cat, Cat, Cat, CAT, cats, dog, 猫, Kat

Nobody programmed any of that. The days of the week cluster together, C types cluster together, and the Chinese character for cat (猫) sits beside the English word. Training arranged the vectors this way because it made predicting the next token easier.

What an embedding does not hold yet

• The 4,096 dimensions have no names. No column means “is an animal”. Meaning lives in directions through the space, spread across many columns at once — the same way knowledge is spread across the weights in step 1.
• The similarities are modest (0.2–0.4), because this is only the starting point. The vector is the same every time the token appears.
• It carries no context. " bank" gets the same row whether the sentence is about rivers or money. Mixing in context is the job of the layers — steps 3 through 6.

The exit door

At the other end of the model, output.weight (also 32,256 rows × 4,096 columns) runs the process in reverse. The final 4,096-number vector is compared against every row, giving one score per vocabulary entry: 32,256 scores for “which token comes next.” Turning those scores into a choice is step 4.

How these numbers were produced on a DGX Spark

Reproduce it: every number in this lesson is printed by lessons/02_tokens_and_embeddings.py in the course’s companion code repository, llm-bytes-up-code. Run it with --server to repeat the check against llama.cpp’s tokenizer. The run this lesson quotes is recorded in that repository’s outputs/2026-10-12/, produced from a fresh clone on the Spark.

Everything in this lesson was read on the course’s DGX Spark on 2026-10-12, from the same GGUF file as lesson 1.

• The vocabulary and merges came from the file’s metadata: tokenizer.ggml.tokens (32,256 strings) and tokenizer.ggml.merges (31,757 rules), read with llama.cpp’s gguf-py.
• The tokenizations were first produced by loading that vocabulary and merge list into the Hugging Face tokenizers library as a byte-level BPE, and then checked against llama.cpp’s own tokenizer. That check ran on a llama.cpp server loaded with this model on the Spark, using its /tokenize endpoint.
  • One example differed, and the table above shows llama.cpp’s answer. For x * 2, the reconstruction produced *, , 2, while llama.cpp produced , *, , 2.
  • The cause is the pre-tokenizer. Before any merges run, text is split into chunks by a regular expression, and the reconstruction used the generic GPT-2 one rather than this model’s. Same vocabulary, same merges, different chunk boundaries — so a slightly different result.
  • Every other example matched exactly, including the emoji, 猫 and the runes.
  • The check also caught a copying error in the first draft of the DGX Spark row, whose ID list had been taken from a longer sentence.
  • The lesson in the lesson: a tokenizer is the vocabulary and the exact procedure. Two tools with identical vocabularies can still disagree.
• The embedding rows were dequantized by hand from Q8_0, using the block layout from lesson 1: each 4,352-byte row is 128 blocks of a 2-byte scale plus 32 int8s.
• The nearest neighbours come from computing cosine similarity against all 32,256 rows. That fits easily in the Spark’s 128 GB; the full table is about 0.5 GB as FP32.