Tokens

Text → numbers the model can see

Sources: Neural Machine Translation of Rare Words with Subword Units — Sennrich et al., 2016; OpenAI token-counting guide

A neural network can only operate on numbers, not text. So before any actual model work happens, every prompt you type goes through a small program that turns text into a list of integers. That program is called a tokenizer, the chunks of text it produces are called tokenstokenThe atomic unit of text the model sees. Roughly a word-fragment — “tokenization” is a piece of text → list of token IDs.See in glossary →, and the fixed list of all possible tokens the model knows about is called its vocabularyvocabularyThe fixed set of tokens a model knows about. Modern LLMs have ~32k–200k entries.See in glossary →.

For modern LLMs the vocabulary has somewhere between 32,000 and 200,000 entries. Each entry has an integer index (its token IDtoken IDAn integer index into the vocabulary that uniquely identifies a token.See in glossary →) and that integer is what flows into the rest of the network. From the model’s point of view, your prompt is not “Hello, world” but [9906, 11, 1917].

Why not just use characters? Or whole words?

If we tokenized one character at a time, every prompt would be very long. A 500-word email might become 3,000 tokens, and since the cost of inference grows with sequence length, that gets expensive fast. The model would also have to learn that c, a, t next to each other means something, instead of being told “this is the word cat” up front.

If we tokenized whole words, common words would be easy, but English alone has hundreds of thousands of them. A word the tokenizer had never seen (a typo, a new product name, a piece of code) would typically be mapped to a generic “unknown” token, losing its spelling and identity. Worse, modern LLMs are expected to handle every major language at once: a vocabulary that covered the words of English, Mandarin, Spanish, Hindi, Arabic, Japanese, Russian, and the other 90+ languages users actually type would balloon into the millions of entries, and still miss every word never written down before.

Many modern LLMs use subword tokenization. In a byte-level BPE tokenizer such as cl100k_base, common words may get a single token, less common words split into pieces, and arbitrary text remains encodable by falling back to byte-level pieces. Other tokenizers use related algorithms such as WordPiece or Unigram, so their exact behavior differs.

Try it

Below is a real tokenizer running in your browser: cl100k_base, a 100,000-token encoding used by several OpenAI models, including GPT-4 and GPT-3.5 Turbo. It is not the tokenizer for every OpenAI model (GPT-4o, for example, uses o200k_base) but it is a useful concrete example. Type anything, or pick one of the sample inputs. Watch how the same text gets chopped up into different numbers of tokens depending on whether it’s common English, code, a long word, or another language.

A few things worth noticing as you play with it:

• Common words are often one token. “The”, “and”, “world”: each is a single chip. Their numerical IDs are just indices into the vocabulary; they do not encode meaning.
• Rare words get split. Try “antidisestablishmentarianism”: it breaks into 5–6 pieces. The model still sees a meaningful sequence; it just has to do a bit more work.
• Code token counts are tokenizer-dependent. Punctuation, whitespace, and unusual identifiers can create more token boundaries, while common pieces such as def and return are single tokens in cl100k_base. Compare the actual count instead of assuming code always uses more tokens than prose.
• Token counts vary across languages. A tokenizer may split some scripts or languages more finely than English, particularly when it lacks common character sequences. But this is tokenizer-specific: modern multilingual vocabularies can encode many non-Latin strings efficiently. Compare the token count for the text you are using rather than assuming all non-English prompts cost more.
• Emoji and rare Unicode can hit the byte fallback: one token per byte of UTF-8.

What the model actually receives

By the end of the tokenizer’s work, your prompt (a string) has become a flat list of integers, each between 0 and vocab_size - 1. Those integers will be the very first thing the neural network sees. The model has no idea they used to be letters; as far as it knows, the world is just a sequence of integer indices.

Of course, integers alone don’t capture meaning. The token ID for “king” and the ID for “queen” are just two arbitrary numbers; they don’t tell the network that those words are related. The next step is to turn each integer into a much richer object that can carry information about meaning: a vector. That’s the topic of the next section: embeddings.