Text becomes tokens

Before a language model does anything with your message, the text is cut into pieces called tokens. Each token is then swapped for a number, its ID in the tokenizer's vocabulary. That list of numbers is what the model actually receives.

One sentence, six tokens

Take the sentence Tokenization is surprisingly simple. In OpenAI's GPT-5.x tokenizer it becomes six tokens:

#	Token	ID
1	Token	4421
2	ization	2860
3	␣is	382
4	␣surprisingly	50999
5	␣simple	4705
6	.	13

A few things stand out:

• "Tokenization" splits in two. Common words often get one token. Longer or rarer ones get broken into familiar chunks.
• The space travels with the next word. ␣is and is are different tokens, so spacing changes the split.
• Punctuation counts. The full stop is a token of its own.

The IDs belong to this one tokenizer. The older GPT-4 tokenizer gives the same sentence different IDs: 3404, 2065, 374, 29392, 4382, 13.

Nobody writes these splits by hand

GPT-style tokenizers learn their vocabulary with byte-pair encoding (BPE). Training starts from small pieces and repeatedly merges the most frequent pair it finds in a large body of text. Do that enough times and common chunks, like ization, end up as single tokens.

Other model families use different methods (WordPiece in BERT, Unigram in T5), so the same sentence can split differently depending on the model.

The merge animation in the video is a simplified illustration of the idea behind BPE, not the real merge order of any production tokenizer.

Why it matters to you

Context limits and API prices are counted in tokens, not words. In English, a token is about three-quarters of a word, so 100 tokens is roughly 75 words. That ratio is only an estimate: it changes with the language and the model.

You can try it yourself on OpenAI's tokenizer page. Paste in a sentence and watch where it splits.