What Is a Token, Anyway?

The rate card is printed per million tokens, and the natural assumption is that a token means a word. It is not. A token is a piece of text the model’s tokenizer treats as one unit — a whole word sometimes, part of a word other times, occasionally a single character or a punctuation mark with its neighbours. The tokenizer decides, and it decides differently across languages, formats, and models.

The practical consequence: the same sentence can tokenize to different counts on different models, and the number on the invoice is the tokenizer’s count, not your word count. The price-list post urges reading the fine print — here, the fine print is the tokenizer.

This post is the mechanics: how tokenization works, why it varies, and how to estimate your own numbers without guessing.

By frequency. Tokenizers are built from large corpora: text pieces that appear often — common words, common word parts, common punctuation patterns — become single tokens, and rarer text is cut smaller. “The” is one token; a rare technical term can be several. The tokenizer optimises for the average of the text it was trained on, and your text may not be average.

The method is byte-pair encoding, and the intuition is compression: the tokenizer is a dictionary of frequent pieces, and any text is expressed as a sequence of dictionary entries. The dictionary is fixed per model — built once, before training — which is why counts are stable for a model and differ between models.

And the dictionary is public. Tokenizer tools exist because the count is deterministic: same text, same model, same number, every time. Estimation is arithmetic, not divination.

Because the invoice counts tokens, and the tokenizer sets the count. Three variables move it. Language: dictionaries trained mostly on English cut English efficiently and other languages less so — a sentence in a poorly covered language can cost several times its English equivalent. Format: code, JSON, tables are token-hungry, punctuation-heavy text burning many small tokens. Vocabulary: specialised terms the dictionary has never seen get chopped into pieces.

Same content, same model — the count can vary by a factor of several across those three. The long-context crossover math and the cost-per-task framework inherit the variance: token count feeds every cost calculation on this blog, and the tokenizer is what keeps the input honest.

Measure, do not assume. Run your actual prompts through the tokenizer before budgeting — the calculator and the model library work in tokens, and the only way to feed them honest numbers is to count your own text. A counted prompt is a fact; an estimated one is a guess wearing a number.

Count the whole request, not just the question: system prompt, examples, tool schemas, conversation history — everything riding along is tokens, and the prompt-caching post covers which parts can be made cheaper. Count both directions too: input and output price differently on most models, and the split sets the blended rate.

Then re-count when the prompt changes. A prompt is a living document; its token count deserves review with the same seriousness as its quality.

Two taxes worth knowing by name. The language tax: serving users in a language the tokenizer covers poorly makes your per-answer cost structurally higher than an English product’s — and the fix is not a cheaper model but a tokenizer-aware choice. Some models tokenize multilingual text far better than others, and the difference shows in the count before a dollar is spent.

The format tax: structured output is expensive to tokenize. JSON with its braces and quotes, code with its punctuation, tables with their spacing — all of it costs more tokens than prose carrying the same information. Getting reliable JSON is its own discipline; the cost side is that reliability carries a token price, and it is worth knowing before the invoice explains it.

When the workload is small enough for the difference between an efficient tokenizer and a wasteful one is a rounding error — a hobby project, an internal tool, a prototype. The count is a curiosity there, and optimising it is premature.

Tokens also stop mattering when a larger cost dwarfs them: engineering hours spent tuning a prompt that runs a few times a day, or quality lost by compressing a prompt until it breaks. The cheapest request is the one that can wait; the cheapest token is sometimes the one you did not spend an afternoon saving.

The honest hierarchy: correctness first, cost per completed task second, token count third. The tokenizer is worth understanding because it is the unit of the bill — not because it is the point of the product.