How LLM Tokenization Actually Works

Every prompt you send is chopped into tokens before the model sees a single parameter, and almost every confusing LLM behavior traces back to where those cuts land. Costs, context limits and a whole genre of "the model can't do basic things" bug reports all live here.

The canonical example: ask a model how many times the letter r appears in "strawberry" and it often gets it wrong. Tokenized with OpenAI's cl100k_base vocabulary, "strawberry" is not a word, it is three pieces: str, aw, berry. The model was never shown the letters.

What a Token Actually Is

A token is a byte sequence that appears often enough in training data to earn its own entry in a fixed vocabulary, produced by byte pair encoding: start from individual bytes and repeatedly merge the most frequent adjacent pair until the vocabulary is full.

Two consequences matter. Tokens are byte sequences, not characters, so a token can end halfway through a multi-byte character. And the vocabulary is frozen at training time, so text that looks like the training data compresses well and text that does not gets shredded into fragments.

Building a Real Tokenizer

You can reason about this or you can measure it. OpenAI publishes its vocabularies, so a working tokenizer is about 60 lines. Node 18 or newer, no dependencies.

js
Copy
// tokenize.mjs - Node 18+, no dependencies. Run: node tokenize.mjs
const ENCODINGS = {
  cl100k_base: "https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken",
  o200k_base: "https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken",
}

// The pre-tokenizer splits text before BPE ever runs. This is why " dog" and "dog" differ.
const PATTERNS = {
  cl100k_base: /(?:'[sdmt]|'ll|'ve|'re)| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+/gu,
  o200k_base:
    /[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+/gu,
}

async function loadRanks(name) {
  const text = await fetch(ENCODINGS[name]).then((r) => r.text())
  const ranks = new Map()
  for (const line of text.split("\n")) {
    if (!line) continue
    const [b64, rank] = line.split(" ")
    ranks.set(Buffer.from(b64, "base64").toString("latin1"), Number(rank))
  }
  return ranks
}

// Standard byte-level BPE: repeatedly merge the adjacent pair with the lowest rank.
function bpe(piece, ranks) {
  if (piece.length === 1) return [piece]
  const parts = [...piece]
  for (;;) {
    let bestRank = Infinity
    let bestAt = -1
    for (let i = 0; i < parts.length - 1; i++) {
      const rank = ranks.get(parts[i] + parts[i + 1])
      if (rank !== undefined && rank < bestRank) { bestRank = rank; bestAt = i }
    }
    if (bestAt === -1) return parts
    parts.splice(bestAt, 2, parts[bestAt] + parts[bestAt + 1])
  }
}

const makeEncoder = (ranks, pattern) => (text) => {
  const out = []
  for (const [piece] of text.matchAll(pattern)) {
    const bytes = Buffer.from(piece, "utf8").toString("latin1")
    if (ranks.has(bytes)) { out.push(bytes); continue }
    out.push(...bpe(bytes, ranks))
  }
  return out
}

const [cl, o200] = await Promise.all([loadRanks("cl100k_base"), loadRanks("o200k_base")])
const encCl = makeEncoder(cl, PATTERNS.cl100k_base)
const encO2 = makeEncoder(o200, PATTERNS.o200k_base)

console.log(`sanity: cl100k "hello world" -> ${encCl("hello world").length} tokens ${JSON.stringify(encCl("hello world"))}`)

const SAMPLES = [
  ["strawberry", "strawberry"], ["leading space", " strawberry"],
  ["number 2026", "2026"], ["number 1234567", "1234567"], ["price", "$1,234.56"],
  ["uuid", "550e8400-e29b-41d4-a716-446655440000"],
  ["english", "The quick brown fox jumps over the lazy dog."],
  ["hindi", "तेज भूरी लोमड़ी आलसी कुत्ते के ऊपर से कूदती है।"],
  ["japanese", "素早い茶色のキツネが怠惰な犬を飛び越えます。"],
  ["json", '{"userId":123,"isActive":true}'],
]
for (const [label, text] of SAMPLES) {
  const a = encCl(text), b = encO2(text)
  const split = a.map((t) => Buffer.from(t, "latin1").toString("utf8")).join("|")
  console.log(`${label.padEnd(15)} chars ${String([...text].length).padStart(3)}  cl100k ${String(a.length).padStart(3)}  o200k ${String(b.length).padStart(3)}  ${split.slice(0, 40)}`)
}

Run on September 14, 2026 with Node 25.5.0:

text
Copy
sanity: cl100k "hello world" -> 2 tokens ["hello"," world"]

sample              chars  cl100k  o200k  cl100k split
--------------------------------------------------------------------------------
strawberry             10       3      3  str|aw|berry
leading space          11       1      1   strawberry
number 2026             4       2      2  20|26
number 1234567          7       3      3  123|45|67
price                   9       6      6  $|1|,|234|.|56
uuid                   36      22     18  550|e|8|400|-|e|29|b|-|41|d|4|-|a|716|-|44|6
english                44      10     10  The| quick| brown| fox| jumps| over| the| la
hindi                  47      50     21  (multi-byte fragments)
japanese               22      32     23  (multi-byte fragments)
json                   30       9     11  {"|userId|":|123|,"|isActive|":|true|}

The sanity line matters: "hello world" producing exactly ["hello", " world"] is what the reference tiktoken library produces, so the implementation is right.

What the Splits Explain

Letter counting. strawberry is str|aw|berry. The model sees three opaque IDs, not ten characters, so counting letters requires it to have memorized the spelling of each fragment rather than to look. That is the entire "how many r's" phenomenon.

The leading space is a different token. strawberry costs 3 tokens; " strawberry" with a leading space costs 1. Trailing whitespace in a prompt genuinely changes what the model is conditioned on, which is why providers warn against ending prompts with a space.

Numbers split arbitrarily. 2026 is 20|26 and 1234567 is 123|45|67. Digits are grouped by frequency, not by place value, so a model doing arithmetic is working with pieces that do not align to columns. $1,234.56 costs 6 tokens.

Identifiers are expensive. A single UUID costs 22 tokens under cl100k_base and 18 under o200k_base. Paste a hundred of them into a prompt and you have spent two thousand tokens on identifiers.

Non-English text costs multiples. The same sentence measured 10 tokens in English, 18 in Spanish, 32 in Japanese and 50 in Hindi under cl100k_base. That is a 5x token tax on Hindi for identical meaning, and since APIs bill per token it is a direct price difference by language. The newer o200k_base vocabulary narrows it considerably, taking Hindi from 50 tokens to 21 and Japanese from 32 to 23, which is one of the least advertised improvements in newer models.

Tokens can split characters. The Hindi and Japanese splits render as fragments because individual tokens end mid-character, cutting a multi-byte UTF-8 sequence in half. This is why streaming APIs sometimes emit a broken character until the next token arrives.

Practical Consequences

1. Never estimate cost by character count for non-English text. Measure it. A 5x difference does not survive a rule of thumb.
2. Do not put raw identifiers in prompts you pay for. Replace UUIDs with short indexes in the prompt and map them back afterwards.
3. Stop asking models to do character-level work. Counting letters, reversing strings and checking spellings are jobs for code, because the model is not looking at characters.
4. Do not end a prompt with a trailing space. It changes tokenization of whatever follows.
5. Re-measure when you change models. Vocabularies differ, and as the JSON row shows, a newer vocabulary is not uniformly cheaper: that sample cost 9 tokens under cl100k_base and 11 under o200k_base.

Conclusion

Tokenization is the layer where a model's input stops being text, and most of its apparently stupid behaviors are artifacts of that boundary rather than failures of reasoning. The vocabularies are public, the algorithm is 60 lines, and running it on your own prompts takes a minute. Do that before your next argument about why the model cannot count.

Related DevToolLab Tools

• AI Token Counter - count tokens for a prompt across model families without wiring up a tokenizer library first.
• LLM Token Cost Calculator - turn those token counts into a per-request price before a multilingual feature reaches production.
• Unicode Inspector - see the code points and UTF-8 bytes underneath text, which is the layer BPE actually operates on.
• Character Counter - compare characters against tokens on the same string to see how far apart the two measures drift.

Related Guides

• Prompt Caching Guide - once you know what a token costs, caching is how you stop paying for the same prefix repeatedly.
• Context Engineering Guide - budgeting a context window once you can measure what goes in it.
• Best Embedding Models and APIs - the same tokenization constraints applied to retrieval rather than generation.
• LLM Evals Guide - measuring output quality after you have controlled the input.