Tokenization
Definition. The process of splitting text into the subword units a model actually consumes. Modern LLMs use byte-pair encoding or a variant such as SentencePiece, learning a vocabulary of roughly 30,000 to 200,000 merges. A token is typically three to four characters of English but can be a single character, a whole common word, or a fragment of a non-Latin script. Every cost, limit and latency figure in an AI system is denominated in tokens, not words.
How it works. Count tokens before you send a request, using the provider's tokenizer library rather than an estimate. Budget the system prompt, the retrieved context, the conversation history and the reserved output separately. When you truncate, truncate at token boundaries and preserve the most recent turns plus the system prompt. For non-English content, measure the ratio empirically: the same sentence in Japanese or Hindi can consume two to three times the tokens of its English equivalent.
Example. A team budgets a feature at 4,000 tokens per request. Their system prompt is 1,200 tokens, retrieved context 2,000, history 600, leaving 200 for output. In testing with English content the average request uses 3,100 tokens. After launching in German and Japanese, average usage rises to 4,600 and requests start failing on the output cap. The fix is a per-locale context budget that shrinks retrieved chunks for high-token-ratio languages rather than a larger model.
Use case. Tokenization drives cost forecasting, context-window planning, chunk sizing in retrieval pipelines, and the design of output limits. It also explains odd model behaviour: a model may struggle to reverse a string or count letters because it never sees individual characters, only merged subword units.
Warnings. Never estimate tokens as words multiplied by 1.3 for a multilingual product. Never split text mid-token when chunking, and never assume a character limit in your UI maps to a token limit in the API.
Limitations. Tokenizers are model-specific, so a token count from one model is wrong for another. They handle whitespace, emoji and code inconsistently, and rare identifiers such as UUIDs or base64 blobs tokenize very inefficiently, sometimes at one token per character, which can triple the cost of a payload that looks small.