The tokenizer shipped before any language model weights. Multilingual tokenizers trained on Latin-heavy web crawls routinely shatter Assamese words into byte-level noise. A tokenizer built for English averages roughly 1.2 tokens per word. Feed the same imported tokenizer Assamese text, and it falls back to raw UTF-8 bytes, turning a single word like প্ৰযুক্তি (technology) into eight or nine meaningless fragments.
We measured this directly. The Axomiya Tokenizer was trained on 162 million tokens drawn from the IndicCorpV2 Assamese split (444.6 million characters, 3.37 million lines, 68 million words), using a script-aware Unigram model capped at a 32,000 token budget. Training completed in about 5 minutes on an optimized Rust backend.
Before training, we corrected a major source of noise in the corpus: Bengali and Assamese share deep Unicode overlap, and digitized Assamese text is full of Bengali character substitutions from early keyboard software, most commonly Bengali ra (U+09B0) appearing where Assamese ra (U+09F0) belongs. Correcting this and related punctuation artifacts rescued 386,175 lines, about 11.44% of the entire dataset.
We also ran an ablation on normalization strategy: NFC (lossless canonical composition) against NFKC (lossy compatibility folding). Counterintuitively, the strictly lossless NFC standard came out very slightly ahead, improving compression by 0.16% across a 50,000 sentence holdout. That told us the efficiency gains come from the script-aware architecture itself, not from lossy character folding, so NFC is what shipped to production.
MuRIL compresses further, but only by dedicating close to 200,000 vocabulary slots to Indic scripts, an embedding layer that monopolizes VRAM for anyone training a foundational model from scratch. At 32,000 tokens, the Axomiya Tokenizer stays lean enough to train from scratch while still cutting Llama 3’s fertility by more than 3.5× and Gemma’s by nearly half.
What this means in practice
Fertility is how many tokens a piece of Assamese becomes. That decides how much text fits in a context window and how long generation takes.
- Token volume: at a price of ₹210 per million tokens, a 1 million word Assamese workload would be about ₹947 with Gemma’s tokenizer and ₹504 with the Axomiya Tokenizer, a 46.7% reduction. Navdyut deploys on your machines, so this is a measure of token volume, not an invoice.
- Context window: a fixed context window holds less Assamese text when each word costs more tokens. Lower fertility means more of a document, or more retrieved passages, actually fit.
- Latency: attention cost scales with the number of tokens, so fewer, more meaningful tokens means faster generation.
Axomiya 31B is trained with this vocabulary. Router uses a separate 8,192-token vocabulary, recorded in its own note.