We didn't scrape the internet. We scanned the shelf.
Most Assamese text on the open web is thin, repetitive, or badly OCR'd. So we built our own corpus the slower way: scanning printed sources page by page and cleaning what came out.
Internet-scale data has an internet-scale problem.
Common web crawls are the default corpus for most language models. For a low-resource language like Assamese, the web is thin, and what exists is skewed toward short, repetitive, machine-translated, or informal text. That's a poor foundation for a model meant to write and reason in formal, literary, and technical Assamese.
- Published books and literary works
- Magazines and periodicals
- Newspapers, current and archival
- Government publications and gazettes
- Educational material and textbooks
Cleaner grammar, wider vocabulary, real editorial register, and coverage of formal Assamese that web text rarely contains. It also means a much smaller share of duplicate or near-duplicate content to filter out before training.
From scanned page to training token.
Scan & extract
Printed pages are digitised and run through our own OCR line, the same vision models behind Lipi Print and Lipi Hand.
Language identification
Assamese and Bengali are distinguished using script-level heuristics (most reliably the ৰ/ৱ distinction), backed by a fastText classifier.
Quality filtering
A perplexity model trained on Assamese Wikipedia flags low-quality or corrupted OCR output before it enters the corpus.
Deduplication
Document- and paragraph-level MinHash deduplication removes reprints and repeated boilerplate across sources.
One language. Multiple registers.
Explore representative, editorially written examples of the registers the corpus is designed to cover. These are previews, not excerpts from licensed training documents.
Published literature
সাহিত্যই কেৱল সময়ৰ কথা নকয়; ই মানুহে নিজৰ পৃথিৱীখন কেনেদৰে বুজিছিল, তাৰো সাক্ষ্য বহন কৰে।
Eri is our proof of process.
We open-sourced Eri so the low-resource NLP community can see what training from scratch on a scanned, curated corpus actually produces, before we scale the same process to Muga and beyond.