NVIDIA’s 2.8-Trillion-Token Indic Corpus: Uses and Checks

NVIDIA and AI4Bharat have announced Nemotron-pretraining-Indic-v1, containing roughly 2.8 trillion tokens across 22 Indian languages, and an open data generation pipeline. Practical uses include testing training data mixtures by language and reusing the generation process. Corpus size alone does not establish quality or improved model performance.

Key Points#

• The corpus includes 2.601 trillion native-script tokens and 224.6 billion romanized tokens, measured with the Gemma-4 tokenizer.
• All 22 languages have native-script data, and 12 also have romanized text derived from translations that passed evaluation.
• The pipeline combines document translation, automated evaluation, and language-specific processing, with feedback-based revision for the 12-language group.
• The reported benchmark improvements come from training on both synthetic data and naturally written Indic text; this corpus’s individual contribution has not been isolated.

What the Corpus Contains#

The corpus consists of translations of English material selected from 15 Nemotron-CC-v2 subsets, including web documents, question-and-answer content, mathematics, and code. “Native script” describes the writing system, not the text’s origin. These records remain translations from English.

Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu have both native-script and romanized data. Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi have only native-script data. Romanized text represents Indian languages in Latin characters; it is not English.

The total includes translations and different-script versions derived from shared source documents, as well as data that passed evaluation after revision. It is neither a count of unique English source documents nor a recommended training token count. NVIDIA’s October 16, 2026 announcement describes the corpus as “coming soon,” so check availability and licensing before planning training.

How Generation Works#

For the 12-language group, NeMo Data Designer uses Gemma-4-31B-it to translate entire documents. Prompts distinguish text requiring translation from code, equations, URLs, and formatting that must be preserved. The same model then evaluates each translation against the English source using a rubric covering meaning, fluency, terminology, style, and formatting.

Outputs must score at least 75 out of 100. This is a rubric-based threshold, not 75% accuracy. Lower-scoring translations undergo feedback-based revision and reevaluation. Revision made 438.4 billion tokens eligible for inclusion, and these are included in the corpus total. Some accepted translations are then romanized and checked against their native-script inputs.

The remaining 10 languages use Bodhan AI’s Indic Translate, followed by Gemma-4-31B-it evaluation at the same threshold, without feedback-based rewriting. Long Santali documents are processed in overlapping chunks, while Manipuri is translated into Bengali script before conversion to Meitei script. Source identifiers are preserved throughout processing.

What the Results Show#

Nemotron 3.5 Super VL underwent continued pretraining with a budget of 100 billion tokens. Of these, 10 billion used a mixture of synthetic data and naturally written Indic text covering 12 languages. The remaining 90% preserved the relative proportions of the previous training data mixture.

Across this training stage, scores rose by 4.98–9.82 percentage points on five reported Indic knowledge and reasoning benchmarks: MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy, and Indic GSM8K. These results support the combined training approach. They do not show how much improvement came from synthetic data alone.

Controlled corpus validation also covers less than the announced release, examining data from the 12-language group that passed its initial evaluation. Evaluation of data accepted after revision and of the remaining 10 languages remains future work. Treat the results as grounds for experimentation, rather than predictions of benefits for every model.

How to Run Useful Trials#

First, choose the languages and scripts your use case requires. Configuration names include language, script, and region: hi-Deva-IN denotes Hindi in Devanagari, while hi-Latn-IN denotes romanized Hindi.

Proceed in this order.

1. Confirm that the repository is available, review the dataset card and license, and pin the revision you will use. 2. Stream samples with Hugging Face Datasets and check meaning, script consistency, and preservation of technical content. 3. Compare training data mixtures with a fixed total token budget, recording the proportions of natural and synthetic data and of native-script and romanized text. 4. Measure per-language performance, retention of English capabilities, and language quality while ensuring that shared source documents do not cross training and evaluation splits.

To generate domain-specific material, use the Nemotron Bharat translation plugin for NeMo Data Designer. Start with a small document sample and have native-speaking reviewers evaluate it before scaling up. The plugin produces candidates and scores; your acceptance policy determines what goes into training.

Quality Checks Needed#

Automated scores are useful filters, but the same model handles translation and evaluation in the 12-language workflow. Human review of sampled outputs supplements the scores, though its coverage varies by language. The 10-language workflow needs broader validation by native speakers.

Translation also inherits the limitations of the source material. It cannot fully supply the local knowledge, cultural context, and linguistic expression found in naturally written Indic text. Combine it with appropriately sourced natural data and evaluate the mixture for your use case.

Finally, matching source IDs removes processing duplicates, not semantically similar passages. Content duplication requires separate checks, especially when the same source appears in multiple languages or scripts.

Next Steps#

Choose one target language and script, and prepare a 100-document review set. Have native-speaking reviewers check meaning, omissions, and technical formatting, then decide on your training data mixture.

Source: https://huggingface.co/blog/nvidia/scaling-indic-pretraining