Navdyut-240M is not an Assamese language release. It’s an architecture proof-of-concept, the foundation the rest of the suite’s scaling work stands on. Where most small models lean on trillion-token web scrapes to compensate for limited parameter budgets, we asked whether a “wide and shallow” design, paired with an aggressively compressed 8,192 token vocabulary, could redistribute that saved parameter mass into a wider hidden state instead. Shrinking the vocabulary from a standard 50k down to 8,192 freed up roughly 108 million parameters, letting us push the hidden dimension to 1,280 while keeping total size under 250M.
We trained on a deliberately narrow, synthetic diet: 14.4 billion tokens of structured logic and math reasoning traces plus curated Python source, no open web crawl. We also used Maximal Update Parametrization to carry hyperparameters found on a 31M proxy model straight up to the 240M scale without a fresh search, and square-root batch scaling to keep training stable as batch size grew across the cluster.
| Step | Epoch | CE Loss | Grad Norm |
|---|---|---|---|
| 5,000 | 0.17 | 3.842 | 0.81 |
| 20,000 | 0.68 | 2.910 | 0.65 |
| 40,000 | 1.37 | 2.501 | 0.52 |
| 60,000 | 2.05 | 2.154 | 0.44 |
| 80,000 | 2.73 | 2.011 | 0.40 |
| 105,000 | 3.59 | 1.928 | 0.38 |
The loss curve is smooth throughout, no spikes, gradient norms stay under 1.0 the whole run. That’s the actual result worth reporting: the parameter redistribution and scaling method are mathematically sound and stable, which is what this experiment set out to prove.
What this model can’t do yet
In the interest of an honest release record: on general reasoning benchmarks, Navdyut-240M scores near random on ARC-Easy and Winogrande, and 0.0% on GSM8K, well behind Pythia-410M and SmolLM-360M, both trained on two to three orders of magnitude more data. On HumanEval Python perplexity, it lands at a cross-entropy loss of 4.58, meaningfully behind those same models. We read this as expected: a 14.4B token diet is enough to prove the architecture holds together, not enough to build real capability on top of it.
Next steps are scaling the same compressed-vocabulary approach to 1.5B parameters, and expanding the context window from the current 256 tokens to something usable for actual retrieval or multi-file code context. Neither of those has happened yet.
Deep architecture documentation stays on the Models page. This post is the release record.