Router is the 248M instruction-tuned model in the dense suite. It is not the Assamese language model; that is Axomiya 31B, which uses the separate 32K Axomiya Tokenizer. Router’s own vocabulary is 8,192 tokens. The note below is the training record for this small dense model: a wide and shallow design. Shrinking the vocabulary from a standard 50k down to 8,192 freed up roughly 108 million parameters, letting us push the hidden dimension to 1,280 while keeping total size under 250M.
We trained on a deliberately narrow, synthetic diet: 14.4 billion tokens of structured logic and math reasoning traces plus curated Python source, no open web crawl. We also used Maximal Update Parametrization to carry hyperparameters found on a 31M proxy model straight up to the 248M scale without a fresh search, and square-root batch scaling to keep training stable as batch size grew across the cluster.
| Step | Epoch | CE Loss | Grad Norm |
|---|---|---|---|
| 5,000 | 0.17 | 3.842 | 0.81 |
| 20,000 | 0.68 | 2.910 | 0.65 |
| 40,000 | 1.37 | 2.501 | 0.52 |
| 60,000 | 2.05 | 2.154 | 0.44 |
| 80,000 | 2.73 | 2.011 | 0.40 |
| 105,000 | 3.59 | 1.928 | 0.38 |
The loss curve is smooth throughout, no spikes, gradient norms stay under 1.0 the whole run. That’s the actual result worth reporting: the parameter redistribution and scaling method are mathematically sound and stable, which is what this experiment set out to prove.
What this model can’t do yet
In the interest of an honest release record: on general reasoning benchmarks, Router scores near random on ARC-Easy and Winogrande, and 0.0% on GSM8K, well behind Pythia-410M and SmolLM-360M, both trained on two to three orders of magnitude more data. On HumanEval Python perplexity, it lands at a cross-entropy loss of 4.58, meaningfully behind those same models. We read this as expected: a 14.4B token diet is enough to prove the architecture holds together, not enough to build real capability on top of it.
Next steps are scaling the same compressed-vocabulary approach to 1.5B parameters, and expanding the context window from the current 256 tokens to something usable for actual retrieval or multi-file code context. Neither of those has happened yet.