What is Token?

Why It Matters

Understanding tokens matters because nearly every practical constraint and cost in working with an AI model, pricing, context window size, response speed, is measured in tokens rather than the words or characters people naturally think in. That gap between word count and token count means a document's actual processing cost or context usage can be a meaningful surprise if someone estimates based on word count alone. Different tokenizers split text differently, so the same document can use a different number of tokens depending on which model processes it, which matters when comparing cost or capacity across providers. It's also the foundation other AI metrics build on, like token efficiency, which measures how much useful output a system produces per token spent. Anyone budgeting for AI usage or designing prompts needs to think in tokens, not words, to get an accurate picture of cost and capacity.

Example

The sentence 'AI models process text as tokens' breaks into roughly 7 tokens depending on the tokenizer used, since some words split into multiple tokens and some short words merge with punctuation. A 1,000-word document might use approximately 1,300 tokens, and if an AI service charges per token, that difference between word count and token count directly affects the actual cost of processing that document, which is why token count, not word count, is the metric used for API billing and context window limits.

Frequently Asked Questions

• Is a token always equal to one word?

  No, a token can be a whole short word, part of a longer word, or even a punctuation mark, which is why token count and word count for the same text are usually different, with tokens typically outnumbering words.

• How is a token different from a context window?

  A token is the individual unit of text being measured, while a context window is the maximum total number of tokens a model can process at once across a given input and output, essentially its working memory limit.

• Why do different AI models count tokens differently?

  Each model uses its own tokenizer, trained on different data and rules for splitting text into subword units, so the same sentence can produce a different token count depending on which model's tokenizer processes it.

• How can you estimate token count without running an actual tokenizer?

  A common rough estimate is about 4 characters or roughly 0.75 words per token in English, though this is only an approximation and the actual count depends on the specific tokenizer used.

• Why does token count matter for cost specifically?

  Most AI providers bill per token for both input and output, so a document or prompt with a higher token count directly costs more to process, regardless of how many words it appears to contain.