Token

A token is a unit into which a model divides text for processing. Depending on the tokenizer, it may represent a word, part of a word, punctuation or another character sequence.

Why models work with tokens instead of characters or words

Tokenisation maps text to a limited vocabulary of numerical identifiers. This allows the model to compose unfamiliar words from smaller existing units.

Different models may tokenise the same text differently.

Why tokens are decisive for cost and speed

Many language-model APIs price input and output tokens separately and at different rates. Long documents therefore increase usage and cost even when the actual question is short.

The number of generated tokens also affects response time because autoregressive models produce output step by step. Reliable estimates depend on the tokenizer of the specific model version in use.

The connection to the context window

A language model’s context window is measured in tokens. Available capacity must account for input, supplied context and, depending on the service, potential output.

A realistic token estimate shows whether documents or histories fit into one request, need to be divided or require a different architecture.