Nearly every token budget in production was built on a single tokenizer and then applied to every model the app talks to. That shortcut is fine right up until it is not, and the point where it breaks is not where most teams expect. This post takes five tokenizers compared on one fixed corpus, runs them on one dated afternoon, and pastes the counts along with the script that produced them.
The audience here is anyone shipping an LLM feature who needs to know how many tokens a request will cost before sending it. Specifically, it answers the question underneath the search: if you count with tiktoken and then route the prompt to Llama or Mistral, how wrong is your number? The measured answer splits cleanly in two, and one half of it contradicts the advice you will usually find.
How These Five Tokenizers Were Compared
Every count below comes from the tokenizer libraries the model vendors publish, running locally against text held in a fixed file. No API was called, so nothing here depends on a provider’s billing endpoint agreeing with its own tokenizer.
Measured 2026-09-16
Windows 11 Pro 25H2 (build 26200), AMD Ryzen 5 8600G (6C/12T), 15.2 GB RAM
Python 3.13.7 (amd64)
tiktoken 0.14.0, transformers 5.17.0, tokenizers 0.23.2, sentencepiece 0.2.2
Encodings: o200k_base, cl100k_base
Model repos, pinned to the revision read on the day:
NousResearch/Meta-Llama-3.1-8B 1f47e50cdbe801ad8a5174156ec3a0655108fb9f
mistralai/Mistral-7B-Instruct-v0.3 c170c708c41dac9275d15a8fff4eca08d52bab71
Qwen/Qwen3-8B b968826d9c46dd6066d109eabc6255188de91218
Token counts do not vary between runs, so the count tables carry no repeat count. A tokenizer is a deterministic function of its vocabulary and its input, meaning the same text at the same revision produces the same integer every time. The throughput section later in the post is the one place timing enters, and that table reports the median of five runs.
These five were chosen because they cover the vocabularies most applications actually hit. Two are OpenAI’s, via tiktoken: o200k_base backs the GPT-4o, GPT-4.1, GPT-5 and o-series models, while cl100k_base backs GPT-4 and GPT-3.5 and still powers most third-party token counters on the web. Those mappings are not guesswork, since tiktoken.model.MODEL_PREFIX_TO_ENCODING in version 0.14.0 lists them directly, and tiktoken.encoding_for_model("gpt-5") returns o200k_base. The other three load through the Hugging Face tokenizers library and represent the open-weight side.
One methodology note matters for honesty. Every count in the main tables uses add_special_tokens=False, which excludes the leading <|begin_of_text|> or <s> marker that some tokenizers add by default. That keeps the comparison to the text itself, and the special tokens get their own measured section later, because in a real chat request they are far from the only extra.
| Tokenizer | Backing library | Vocabulary size |
|---|---|---|
GPT-4o (o200k_base) | tiktoken 0.14.0 | 200,019 |
GPT-4 (cl100k_base) | tiktoken 0.14.0 | 100,277 |
| Llama 3.1 | tokenizers 0.23.2 | 128,256 |
| Mistral v0.3 | tokenizers 0.23.2 | 32,768 |
| Qwen3 | tokenizers 0.23.2 | 151,669 |
Tokenizers Compared: The Full Token Count Table
The corpus holds ten samples covering the text types a real application sends: a support triage prompt, English prose, TypeScript, Python, a minified JSON payload, four non-English prose samples, and a line of identifiers. Every sample is published verbatim later in this post.
| Text (chars) | GPT-4o | GPT-4 | Llama 3.1 | Mistral | Qwen3 |
|---|---|---|---|---|---|
| Support triage prompt (452) | 103 | 103 | 103 | 124 | 110 |
| English prose (505) | 92 | 92 | 92 | 100 | 92 |
| TypeScript source (639) | 160 | 160 | 160 | 217 | 160 |
| Python source (462) | 115 | 116 | 116 | 156 | 117 |
| Minified JSON payload (377) | 148 | 147 | 147 | 200 | 174 |
| German prose (319) | 69 | 91 | 91 | 98 | 91 |
| Chinese prose (101) | 73 | 102 | 75 | 103 | 65 |
| Arabic prose (202) | 61 | 146 | 79 | 180 | 85 |
| Russian prose (298) | 76 | 136 | 89 | 123 | 99 |
| UUIDs, keys and hashes (227) | 140 | 150 | 150 | 202 | 193 |
Read that table row by row and a pattern separates the top half from the bottom half. For English and for code, four of the five tokenizers agree almost exactly. For everything else, they disagree by amounts that would wreck a budget.
The Result That Argues Against the Usual Advice
The common warning is that token counts differ meaningfully between models, so you must never estimate one model’s usage with another model’s tokenizer. On English prose and on source code, the measurement does not support that warning.
English prose came out at 92 tokens on GPT-4o, GPT-4, Llama 3.1 and Qwen3. Not approximately 92 on each, but exactly 92 on all four. The TypeScript sample repeated the result at exactly 160 tokens across the same four, and the Python sample spread across only three integers, from 115 to 117.
That is a maximum disagreement of 1.7% on Python and zero on the other two. Consequently, if your traffic is English text and code, counting with cl100k_base and routing to Llama 3.1 or Qwen3 introduces an error smaller than the rounding in your own cost model. The widely repeated caution is, for that specific traffic, overstated.
Two mechanisms explain the convergence. First, Meta’s Llama 3 model card describes a 128k vocabulary built on tiktoken, and the measured splits bear that out: on English, German and TypeScript, Llama 3.1 matched cl100k_base token for token. Second, byte-level BPE trained on similar web corpora converges on similar merges for the most frequent English and programming-language patterns, because those patterns dominate the training data for all of them.
Mistral is the exception in every row, and the reason is its vocabulary. At 32,768 entries it holds roughly a sixth of what o200k_base holds, so it has far fewer merged multi-character tokens available and falls back to shorter pieces.
Where the Same Text Costs Nearly Three Times as Much
Non-Latin scripts are where the tokenizers stop agreeing, and the spread is not subtle.
| Text | Cheapest | Most expensive | Spread |
|---|---|---|---|
| Arabic prose | GPT-4o, 61 | Mistral, 180 | 2.95x |
| Russian prose | GPT-4o, 76 | GPT-4, 136 | 1.79x |
| Chinese prose | Qwen3, 65 | Mistral, 103 | 1.58x |
| German prose | GPT-4o, 69 | Mistral, 98 | 1.42x |
The Arabic row is the headline. That identical paragraph costs 61 tokens on GPT-4o and 146 on GPT-4, an increase of 139% for switching between two tokenizers from the same vendor. Against Mistral the figure reaches 195%.
Watching a single fragment split shows exactly where the tokens go. The Arabic phrase for “connection pooling” is fifteen characters, and ␣ below marks a token that is or begins with a space:
تجميع الاتصالات
o200k_base 5 tokens ت|ج|ميع|␣الات|صالات
cl100k_base 10 tokens ت|ج|م|ي|ع|␣ال|ات|ص|ال|ات
Llama 3.1 5 tokens ت|جميع|␣الات|ص|الات
Mistral v0.3 13 tokens ␣|ت|ج|م|ي|ع|ال|ا|ت|ص|ال|ا|ت
Qwen3 6 tokens ت|جميع|␣ال|ات|صال|ات
Mistral’s leading ␣ is not an artefact of the fragment. SentencePiece prepends a whitespace marker to the start of any input, so that token appears whether or not the text begins with a space.
Mistral is spending one token per character, which is the worst case a BPE tokenizer can reach. Meanwhile o200k_base has learned multi-character Arabic merges and gets the same content into five. Notably, Llama 3.1 diverges from cl100k_base here despite matching it exactly on English, which is consistent with those extra 28,000 vocabulary entries being spent on non-English text.
Qwen3 produces the only negative in the whole comparison. On Chinese prose it needs 65 tokens where GPT-4o needs 73, making it 11% cheaper than the model with a vocabulary a third larger. For an application serving Chinese, that difference compounds on every request.
Why Code and JSON Punish a Small Vocabulary
Source code separates the tokenizers by a consistent margin rather than a dramatic one. TypeScript cost Mistral 217 tokens against 160 everywhere else, a 35.6% premium. Python and the JSON payload landed on nearly the same figure, at 35.7% and 35.1%.
One line of the TypeScript sample demonstrates the mechanism:
const response = await fetch(`https://api.example.com/v1/invoices/${id}`, {
o200k_base 20 tokens ␣|␣const|␣response|␣=|␣await|␣fetch|(`|https|://|api|.example|.com|/v|1|/in|voices|/${|id|}`,|␣{
Mistral v0.3 26 tokens ␣|const|response|=|await|fetch|(`|https|://|api|.|example|.|com|/|v|1|/|in|vo|ices|/${|id|}|`,|{
The difference is almost entirely whitespace and punctuation. Byte-level BPE tokenizers merge a leading space into the following word, so " response" is a single token. Mistral’s SentencePiece vocabulary handles spacing differently here and splits .example.com into five pieces where the others use three.
The JSON row deserves separate attention, because minified JSON is the densest text most applications send. At 377 characters it cost 148 tokens on GPT-4o, meaning 2.55 characters per token against 5.49 for English prose. In other words, a JSON payload burns roughly twice the tokens per character that prose does, which is the practical argument for trimming fields before they reach a prompt rather than after. If you are returning structured data rather than sending it, the schema-first approach in structured LLM outputs with Instructor and Pydantic keeps the response side tight for the same reason.
This also has a context-window consequence that rarely gets stated. Using the measured 3.99 characters per token for TypeScript on Llama 3.1 and 2.94 on Mistral, a 32k context window holds about 130,700 characters of TypeScript on one and about 96,300 on the other. Same nominal window, 36% more code inside it.
Identifiers Are the Most Expensive Text You Send
The identifier sample is 227 characters of UUIDs, Stripe-style keys and a SHA-256 hash. It cost 140 tokens on GPT-4o and 202 on Mistral, working out to between 1.12 and 1.62 characters per token. That is the worst ratio in the corpus, worse even than Chinese.
A single UUID makes the reason obvious:
9f8b2c14-6d3e-4a71-9c05-2be7f1a8d340 (36 characters)
o200k_base 30 tokens 9|f|8|b|2|c|14|-|6|d|3|e|-|4|a|71|-|9|c|05|-|2|be|7|f|1|a|8|d|340
Mistral v0.3 36 tokens ␣|9|f|8|b|2|c|1|4|-|6|d|3|e|-|4|a|7|1|-|9|c|0|5|-|2|be|7|f|1|a|8|d|3|4|0
Qwen3 35 tokens 9|f|8|b|2|c|1|4|-|6|d|3|e|-|4|a|7|1|-|9|c|0|5|-|2|be|7|f|1|a|8|d|3|4|0
Random hexadecimal has no recurring patterns for BPE to merge, so every tokenizer degenerates toward one token per character. Mistral and Qwen3 reach it almost exactly, splitting all but one of the 36 characters into its own token. As a result, thirty UUIDs pasted into a prompt cost roughly 900 to 1,080 tokens regardless of which model reads them.
The practical lever is to stop sending them. Replacing UUIDs with short sequential indexes in the prompt, then mapping back in your own code, converts a 30-token identifier into a 1-token one. For a retrieval prompt carrying fifty document IDs, that single change frees around 1,400 tokens per request.
The chars/4 Heuristic Is Wrong in Both Directions
Dividing character count by four is the estimate most teams reach for before they wire up a real tokenizer. Measured against all five tokenizers, it is not a conservative approximation. Instead, it is wrong by large margins, and the sign of the error flips depending on the text.
| Text | chars/4 estimate | Error vs GPT-4o | Error vs Mistral |
|---|---|---|---|
| English prose | 126 | +37.2% | +26.2% |
| Support triage prompt | 113 | +9.7% | -8.9% |
| TypeScript source | 160 | -0.2% | -26.4% |
| Russian prose | 74 | -2.0% | -39.4% |
| Arabic prose | 50 | -17.2% | -71.9% |
| Minified JSON payload | 94 | -36.3% | -52.9% |
| UUIDs, keys and hashes | 57 | -59.5% | -71.9% |
| Chinese prose | 25 | -65.4% | -75.5% |
On English prose the heuristic overestimates by 37%, which wastes budget but fails safe. On JSON, identifiers and Chinese it underestimates by 36% to 75%, which is the direction that produces a context-length error in production rather than a slightly padded invoice.
The rule that survives this table is narrow. Roughly four characters per token holds for English prose and for code on the OpenAI and Llama vocabularies, and it collapses everywhere else. Since a real tokenizer call costs microseconds, as the throughput section shows, the heuristic has no performance justification left. For the budgeting layer that sits on top of an accurate count, token counting and budget management for LLM apps covers the enforcement side.
Chat Templates Add Tokens Your Counter Never Sees
Counting the raw string understates a real request, because open-weight models wrap messages in a chat template before tokenizing. Rendering the template and counting the difference gives the overhead exactly.
| Conversation | Llama 3.1 Instruct | Mistral v0.3 | Qwen3 |
|---|---|---|---|
| One user message | 10 | 3 | 8 |
| System plus user | 15 | 5 | 13 |
| System, user, assistant, user | 25 | 8 | 23 |
Here the ranking inverts. Mistral, the most expensive tokenizer on content in every other table, carries the cheapest template by a wide margin, because its format is a pair of [INST] markers rather than per-message role headers. Llama 3.1 spends 10 tokens on a single user message before any content is counted at all.
The rendered strings show why:
Llama 3.1 Instruct
<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{system}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{user}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n
Mistral v0.3
<s>[INST] {system}\n\n{user}[/INST]
Qwen3
<|im_start|>system\n{system}<|im_end|>\n<|im_start|>user\n{user}<|im_end|>\n<|im_start|>assistant\n
The cost scales with turn count, not with message length. Therefore a long-running chat session pays the overhead on every turn it replays, and an agent loop that resends twenty prior messages to Llama 3.1 spends well over 100 tokens on scaffolding alone. Trimming history matters more than trimming wording, which is also the economic case behind semantic caching for LLMs.
How Fast Each Tokenizer Runs
Counting tokens sits on the request path, so its cost is worth knowing. This benchmark concatenates the full corpus forty times into a single 144,000-character string and encodes it, reporting the median of five runs.
| Tokenizer | Median time | Tokens produced | Throughput |
|---|---|---|---|
o200k_base | 5.7 ms | 41,480 | 25.17 M chars/s |
cl100k_base | 8.4 ms | 49,720 | 17.24 M chars/s |
| Mistral v0.3 | 39.5 ms | 60,760 | 3.64 M chars/s |
| Llama 3.1 | 52.3 ms | 44,080 | 2.75 M chars/s |
| Qwen3 | 56.1 ms | 47,440 | 2.57 M chars/s |
tiktoken runs roughly ten times faster than the Hugging Face tokenizers on this input, at 25.17 against 2.57 million characters per second. Both are Rust implementations called from Python, so the gap comes from the algorithm and the per-call overhead rather than from one being interpreted.
Put in request terms, though, both are irrelevant. A 2,000-character prompt takes about 0.08 ms on o200k_base and about 0.8 ms on Qwen3, against a network round trip measured in hundreds of milliseconds. Consequently, the argument for the chars/4 heuristic on performance grounds does not survive contact with the numbers.
Reproduce Every Number in This Post
The corpus and the counting script are below. Install the dependencies first:
python -m pip install tiktoken==0.14.0 transformers==5.17.0 sentencepiece==0.2.2 protobuf
Save the corpus as corpus.py. The four non-English samples are translations of the English one, which keeps content constant across scripts:
PROMPT = """You are a support triage assistant for a SaaS billing platform.
Classify the customer message into exactly one category: billing, bug, feature_request, or account.
Return strict JSON with keys category, confidence, and reason. Do not add commentary.
Customer message:
Hi, I was charged twice for the October invoice (INV-2026-10-4471) on the card ending 4412.
The second charge shows as pending in my bank app. Can you refund it and confirm by email?
"""
ENGLISH = """Connection pooling solves a problem that does not exist until it suddenly does. A single Postgres backend process holds roughly ten megabytes of resident memory, so a pool of twenty connections is unremarkable and a pool of five hundred is a memory incident waiting for traffic. The failure is rarely gradual. Instead, the database accepts connections normally until it crosses the point where the operating system starts reclaiming page cache, and then query latency climbs across every endpoint at once."""
JSON_PAYLOAD = """{"invoice_id":"9f8b2c14-6d3e-4a71-9c05-2be7f1a8d340","customer":{"id":"cus_PQr7XkLm2","email":"dana.whitfield@example.com","country":"DE"},"lines":[{"sku":"SEAT-PRO","quantity":14,"unit_price_cents":2900},{"sku":"ADDON-SSO","quantity":1,"unit_price_cents":19900}],"subtotal_cents":60500,"tax_cents":11495,"total_cents":71995,"status":"open","created_at":"2026-09-14T08:31:07Z"}"""
IDS = """9f8b2c14-6d3e-4a71-9c05-2be7f1a8d340 cus_PQr7XkLm2 sk_live_51MxAbCdEfGhIjKlMnOpQ e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 INV-2026-10-4471 7c9e6679-7425-40de-944b-e07fc1f90ae7 pi_3PqR8s2eZvKYlo2C1aBcDeFg"""
CORPUS = [
("Support triage prompt", PROMPT),
("English prose", ENGLISH),
("Minified JSON payload", JSON_PAYLOAD),
("UUIDs, keys and hashes", IDS),
]
The full ten-sample corpus adds the TypeScript, Python, German, Chinese, Arabic and Russian entries in the same shape. Next, save the counter as count_tokens.py:
import tiktoken
from transformers import AutoTokenizer
from corpus import CORPUS
# Pin revisions so the table stays reproducible after the repos move.
HF_REPOS = {
"Llama 3.1": ("NousResearch/Meta-Llama-3.1-8B", "1f47e50cdbe801ad8a5174156ec3a0655108fb9f"),
"Mistral v0.3": ("mistralai/Mistral-7B-Instruct-v0.3", "c170c708c41dac9275d15a8fff4eca08d52bab71"),
"Qwen3": ("Qwen/Qwen3-8B", "b968826d9c46dd6066d109eabc6255188de91218"),
}
o200k = tiktoken.get_encoding("o200k_base")
cl100k = tiktoken.get_encoding("cl100k_base")
hf = {name: AutoTokenizer.from_pretrained(repo, revision=rev)
for name, (repo, rev) in HF_REPOS.items()}
NAMES = ["GPT-4o", "GPT-4", "Llama 3.1", "Mistral v0.3", "Qwen3"]
def counts(text):
# add_special_tokens=False measures the text itself, not the BOS marker
# some tokenizers prepend. Chat templates are counted separately below.
out = {"GPT-4o": len(o200k.encode(text)), "GPT-4": len(cl100k.encode(text))}
for name, tok in hf.items():
out[name] = len(tok.encode(text, add_special_tokens=False))
return out
print(f"{'Text':<24}{'Chars':>7}" + "".join(f"{n:>14}" for n in NAMES))
for label, text in CORPUS:
c = counts(text)
print(f"{label:<24}{len(text):>7}" + "".join(f"{c[n]:>14}" for n in NAMES))
Run it with python count_tokens.py. On a first run the Hugging Face tokenizers download to the local cache, which takes a few seconds each; afterwards the script finishes in about two seconds. Here is the raw output for the four samples above:
Text Chars GPT-4o GPT-4 Llama 3.1 Mistral v0.3 Qwen3
Support triage prompt 452 103 103 103 124 110
English prose 505 92 92 92 100 92
Minified JSON payload 377 148 147 147 200 174
UUIDs, keys and hashes 227 140 150 150 202 193
To reproduce the chat-template table, render the messages to a string first and then encode without special tokens, since the template already includes them:
messages = [
{"role": "system", "content": "You are a support triage assistant."},
{"role": "user", "content": "I was charged twice for invoice INV-2026-10-4471."},
]
for name, tok in hf.items():
rendered = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
total = len(tok.encode(rendered, add_special_tokens=False))
content = sum(len(tok.encode(m["content"], add_special_tokens=False)) for m in messages)
print(f"{name:<14} total={total:>4} content={content:>4} overhead={total - content:>3}")
Counting tokenize=True output directly is the common mistake here, because recent transformers releases return a dictionary rather than a list of ids, so len() silently yields the number of keys instead of the number of tokens.
When to Use Each Tokenizer for Counting
The decision is not which tokenizer is best, since each one is correct for its own model by definition. Rather, the decision is which tokenizer you can safely count with when you cannot run the target model’s own.
Reach for tiktoken when
- Your traffic is English prose or source code, where four of five vocabularies agreed to within 1.7%
- You need counting on the hot path and are running it per chunk across a large corpus
- You are routing between OpenAI models and Llama 3.1, which matched
cl100k_baseexactly on English, German and TypeScript - You want zero model downloads, since tiktoken fetches a small encoding file rather than a repository
Load the model’s own tokenizer when
- Any meaningful share of your traffic is non-Latin script, where the spread reached 2.95x
- You are targeting Mistral or another small-vocabulary model, which ran about 35% above the field on code in every sample
- You sit close enough to the context limit that a 30% underestimate truncates a request
- Your prompts carry identifiers, hashes or dense JSON, where the vocabularies diverged by up to 44%
Measure the chat template separately when
- You are building agent loops that replay conversation history on every turn
- You enforce a hard token ceiling rather than a soft budget
- You compare costs between a hosted API and a self-hosted deployment of the same weights, which you can stand up following running local LLMs with Ollama
When NOT to Rely on These Numbers
- Your text type is absent from the corpus, since these ten samples do not generalize to Japanese, Hindi, SQL dumps or base64 blobs
- You need billing accuracy rather than budget accuracy, because a provider’s invoice reflects its own server-side counter and may include tokens the client tokenizer never sees
- The model you are targeting is not one of these five, as vocabulary size alone does not predict the count
- You are on a much newer revision of one of these repos, given that vendors occasionally extend a vocabulary between releases
- You are counting images, audio or tool-call payloads, none of which pass through a text tokenizer at all
Common Mistakes with Token Counting
- Using one tokenizer for every model in a router. The error is negligible on English and reaches 195% on Arabic, so the mistake stays invisible until traffic shifts.
- Calling
len(tok.encode(text))with default arguments. Several tokenizers prepend a BOS token, which inflates every count by one and quietly compounds across thousands of chunks. - Estimating with chars/4 and calling it conservative. It overestimated English by 37% but underestimated Chinese by 65%, so it fails in the dangerous direction on exactly the traffic that needs headroom.
- Ignoring the chat template. Content counting misses 10 tokens per user message on Llama 3.1, and that figure grows with every turn replayed.
- Sending raw UUIDs into prompts. At roughly 30 tokens each, fifty identifiers cost around 1,500 tokens before a single instruction is written.
- Trusting a browser token counter. Most are pinned to
cl100k_base, which the table above shows is the wrong vocabulary for every non-OpenAI model once the text leaves English.
A Real-World Scenario: When a Router Crosses a Language Boundary
Consider a mid-sized support platform that classifies inbound tickets with an LLM, handling on the order of hundreds of thousands of messages a month. The team built its budgeting layer around cl100k_base because GPT-4 was the first model they shipped, and a hard cap rejects any request estimated above the context limit. Over several quarters they added a cheaper open-weight model for high-volume tiers and routed a share of traffic to it.
While the product served English-speaking customers, the arrangement held. The estimator and the actual model agreed closely enough that nobody looked, which is the ordinary fate of a component that has never been wrong.
Expansion into a market using a non-Latin script is where the design fails, and it fails asymmetrically. Requests estimated at 146 tokens against cl100k_base might cost 61 on a GPT-4o route, so the cap rejects requests that would have fit comfortably and the team sees an unexplained drop in classification coverage. On a Mistral route the error runs the other way, since the same text costs 180 tokens, and prompts the estimator cleared arrive over the limit and truncate mid-instruction.
The trade-off in fixing it is real rather than free. Loading three model tokenizers at startup costs memory, adds repository downloads to the deploy, and couples the service to Hugging Face availability unless the files are vendored. For a small team the pragmatic middle path is to load the target tokenizer only on routes where the spread was measured to be large, keeping tiktoken for the English and code paths where the four-way agreement was exact. Whatever the routing layer decides, the retry behaviour around a rejected request deserves its own thought, which LLM rate limiting and retry strategies covers directly.
What This Comparison Does Not Measure
No Anthropic tokenizer appears here, because Claude’s vocabulary is not published as a local library. Counting tokens for those models requires the provider’s own token-counting endpoint, so the numbers would not have been comparable with the offline method used throughout.
Output tokens are entirely absent. Every count above is input, and generation length depends on the model’s behaviour rather than on the tokenizer, so nothing here predicts what a response will cost.
These counts also say nothing about money. Token counts multiply by prices that differ per model and change without notice, and a cheap tokenizer attached to an expensive model loses to the reverse. The dated pricing side lives in LLM pricing and cost per task, and the two posts are meant to be read together.
Quality is unmeasured too, which is the most important omission. Fewer tokens is not better tokenization if the splits break words in ways that degrade the model’s understanding, and nothing in a token count reveals that. Mistral’s 32k vocabulary is a deliberate design choice with its own advantages, including a much smaller embedding matrix, rather than merely a worse one.
Finally, the corpus is ten samples of a few hundred characters each. Long documents shift the ratios somewhat, since BPE gets more opportunity to apply long merges, so treat the chars-per-token figures as indicative for prompt-sized text rather than for a book.
Conclusion
With five tokenizers compared on one fixed corpus, the result divides along a clean line. For English prose and source code, GPT-4o, GPT-4, Llama 3.1 and Qwen3 agreed to within 1.7% and frequently to the exact integer, which makes the usual warning about cross-model counting overstated for that traffic. For non-Latin scripts, identifiers and dense JSON, the same tokenizers diverged by up to 2.95x, which makes the warning an understatement.
Mistral v0.3 is the consistent outlier on content, running about 35% above the field on code and up to 195% above on Arabic, and its 32,768-entry vocabulary is the reason. Yet it carries the cheapest chat template at 3 tokens against Llama 3.1’s 10, so the ranking depends on whether your requests are long documents or many short turns.
The recommendation is to route by text type rather than by model. Keep tiktoken for English and code paths, where it is fast and measurably accurate across four vocabularies, and load the target model’s own tokenizer for any path carrying non-Latin script, identifiers or a tight context budget.
The concrete next step takes about ten minutes. Pull a representative day of real prompts from your logs, run them through the counting script above against both cl100k_base and your actual target model, then look at the largest disagreement rather than the average. If that gap sits under 2%, your current estimator is fine and you can stop there. If it does not, token counting and budget management for LLM apps is the next thing to read.
