The tokenizer shipped before any language model weights. Multilingual tokenizers trained on Latin-heavy web crawls routinely shatter Assamese words into byte-level noise. A tokenizer built for English averages roughly 1.2 tokens per word. Feed the same imported tokenizer Assamese text, and it falls back to raw UTF-8 bytes, turning a single word like প্ৰযুক্তি (technology) into eight or nine meaningless fragments.
We measured this directly. Navdyut-Asm-32k was trained on 162 million tokens drawn from the IndicCorpV2 Assamese split (444.6 million characters, 3.37 million lines, 68 million words), using a script-aware Unigram model capped at a 32,000 token budget. Training completed in about 5 minutes on an optimized Rust backend.
Before training, we corrected a major source of noise in the corpus: Bengali and Assamese share deep Unicode overlap, and digitized Assamese text is full of Bengali character substitutions from early keyboard software, most commonly Bengali ra (U+09B0) appearing where Assamese ra (U+09F0) belongs. Correcting this and related punctuation artifacts rescued 386,175 lines, about 11.44% of the entire dataset.
We also ran an ablation on normalization strategy: NFC (lossless canonical composition) against NFKC (lossy compatibility folding). Counterintuitively, the strictly lossless NFC standard came out very slightly ahead, improving compression by 0.16% across a 50,000 sentence holdout. That told us the efficiency gains come from the script-aware architecture itself, not from lossy character folding, so NFC is what shipped to production.
MuRIL compresses further, but only by dedicating close to 200,000 vocabulary slots to Indic scripts, an embedding layer that monopolizes VRAM for anyone training a foundational model from scratch. At 32,000 tokens, Navdyut stays lean enough for resource-constrained hardware while still cutting Llama 3’s fertility by more than 3.5× and Gemma’s by nearly half.
What this means in practice
Fertility isn’t just an academic number, it’s what you’re billed for and how far your context window actually stretches.
- Cost: at roughly ₹210 per million tokens, a 1 million word Assamese workload costs about ₹947 through Gemma’s tokenizer versus ₹504 through Navdyut, a 46.7% reduction.
- Context window: a model with 128,000 tokens of memory holds proportionally less Assamese text under a high-fertility tokenizer. Lower fertility means more of a document, or more retrieved passages in a RAG pipeline, actually fit.
- Latency: attention cost scales with the number of tokens, so fewer, more meaningful tokens means faster generation.
Navdyut-Asm-32k is built to graft onto existing open-source architectures like Gemma or Llama by extending the vocabulary layer, rather than requiring a full retrain from scratch.
Deep model documentation stays on the Models page. This post is the release record.