Found 1 article tagged with "how llm tokenization works"
A model never sees letters. We built a real BPE tokenizer on OpenAI's published vocabularies and measured why strawberry, numbers and Hindi all go wrong.
