Fourteen of sixteen tokenizers beaten.
Including Sarvam‑30B and 105B by 14%, at half the vocabulary. Every script rule validated against fifty million words of real text, and every number here ships with the command that regenerates it.
- Worst case
- 3.23tokens per word · best of 16 tested
- Akshara splitting
- 0.17%19× better than Sarvam‑30B
- Parity
- 1.41×what an English user pays
- Coverage
- 22/22at full round‑trip fidelity
The result
Tokens per word, twenty‑two languages
Measured on 1,024 identical sentences from IN22‑Gen — the same content in every language, which is what makes the comparison legitimate. Lower is better.
Seven of sixteen shown, scaled to the widest bar. Four tokenizers run past it —
Llama‑3 at 8.551, Qwen‑3 at 8.680 and GPT‑4 at 10.008 — and are left off rather than
clipped, since a truncated bar misreports the very thing a bar chart is for.
Sarvam‑105B, Sutra, XLM‑R, BLOOM and Krutrim‑2 sit between 2.45 and 5.46.
Full table in RESULTS.md.
Fairness
What an Indian‑language user pays
The ratio of Indic to English cost. It is the number that reaches a person: how much more they pay, how much longer they wait, how much less context fits — for the same sentence.
Lowest of anything tested. Scaled to GPT‑4; no bar is clipped.
Method
We do not assert what is correct. The corpus does.
Nobody here reads twelve Indic scripts, so no rule was written down as fact. Each one — this codepoint never begins an orthographic unit — was stated as a hypothesis and tested against fifty million words per script. A rule real writers violate is our error, and it is rejected automatically.
137 rules survived. One was rejected, and that one is the argument for the whole method.
Malayalam vowel sign ā — rejected. 375 per million, across 11,495 documents. 254‑ാം is standard ordinal notation, the equivalent of “254th”. Our rule was simply wrong.
A hand‑written rule would have marked it restricted, then counted every ാം token in every tokenizer as a defect — publishing a false claim about other people's work.
Fed hypotheses known to be false, the same corpus rejects seven to ten of ten — so acceptance is a result rather than a rubber stamp. Thresholds are committed and hashed before each run, which makes post‑hoc tuning visible in the git history.
Benchmark design
Fertility rewards deleting text
Our own first run reported a Hindi‑only tokenizer as the best in the world at Santali. It scored 1.09 by encoding Santali to nothing and decoding to nothing. Sarvam‑30B, which handles Santali correctly, scored “worse” at 2.15.
Any leaderboard without a fidelity gate produces that. Ours marks a language not encoded when under 90% of its characters survive a round trip. We found no published intrinsic‑metric work that does this.
Byte‑level vocabularies also have to be decoded before they can be judged. Otherwise Llama‑3 and Qwen score a flawless zero — their vocabularies hold no readable Indic at all.
The trade
What correctness costs
An akshara is what a reader of a Brahmic script perceives as one character. A token beginning with a dependent vowel sign can only ever cut one in half. Every tokenizer measured does this; we removed the possibility rather than penalising it.
| 128k Unigram | Splitting | Fertility |
|---|---|---|
| Unconstrained | 7.87% | 2.422 |
| Constrained | 0.25% | 2.499 |
97% of splitting removed for 3.2% compression. Publishing that price is the point — a correctness claim whose cost is unknown is a slogan.
Findings
Defects in published work
Not criticisms of these teams' intended use. They are the reason a fertility table without a fidelity column misleads.
Google · MuRIL
Discards zero‑width joiner, which destroys Malayalam chillu — ള് becomes ള്, a different letter. 39.1% of sentences survive with content intact.
AI4Bharat · IndicBERT‑v2
Ships no decoder configuration, so decode() returns WordPiece
## markers rather than text. We reassemble manually so its score
measures the tokenizer, not its config file.
AI4Bharat · Sangraha
Encoding damage across the verified splits — 0.4% of Devanagari documents rising to 6.8% of Telugu. One Bengali falsifier decodes to a URL typed on a Bengali keyboard layout and mis‑transcoded. Undocumented before this.
Limits
What we did not win
MuRIL leads on average fertility — 1.980 against 2.122. On its own fourteen‑language set the gap narrows to 1.5% and we lead on parity, but the headline number is theirs. Three attempts to close it failed: more vocabulary saturates, more data bought nothing, and superword tokenization made it worse.
- Every metric here is a proxy. Fertility has been shown not to predict downstream model performance, and neither has morphological alignment. We claim nothing about model quality.
- Santali coverage is thin — 74,000 words against millions for the major languages. Published beside the result rather than averaged away.
- Romanized text is not handled. How a great many people actually type remains the most interesting open problem here, and it is untouched.