NVIDIA and AI4Bharat Detail a 2.8-Trillion-Token Indic Pretraining Corpus, Releasing Soon

NVIDIA and AI4Bharat published on Hugging Face on 16 October 2026 a detailed account of Nemotron-pretraining-Indic-v1: approximately 2.8 trillion tokens across 22 Indian languages, releasing soon, together with the open, GPU-accelerated data-generation pipeline used to build it.

This is a companion to TSN’s coverage of NVIDIA’s Nemotron 3.5 Super VL release (live post 22177), which previewed Indic continued pretraining. The new blog is the corpus-and-pipeline story, not a re-announcement of the model weights.

What is in the corpus

Authors from AI4Bharat (IIT Madras) and NVIDIA say the 22 languages are those listed in India’s Constitution — described as first languages for 97% of Indians, or nearly 1.2 billion people at the 2011 Census. Using the Gemma-4 tokenizer, they report 2,601.0 billion native-script tokens and 224.6 billion romanized tokens. Twelve languages have both native-script and romanized configurations; ten have native-script only (34 language-script configurations in total).

Important framing from the authors: native-script here still means translated from English sources; romanized means Indic text written in Latin script, not an English translation. Document records include translations and script variants of shared sources — they are not counts of unique English documents.

Open pipeline, not only a dump

The team starts from documents in 15 Nemotron-CC-v2 subsets (web, Q&A, maths, code and other specialised English material) and runs translation, quality assessment and targeted repair through NeMo Data Designer. For twelve languages, Gemma-4-31B-it translates whole documents, scores outputs on an MQM-inspired rubric, and retains only outputs scoring at least 75 out of 100. Repair of lower-scoring outputs recovered 438.4 billion tokens already included in the corpus total. For the other ten languages, Indic Translate (Bodhan AI / IIT Madras) handles translation, with Gemma-4 as judge; those workflows produce native-script data only.

Every record is meant to trace back to a curated source document. The open “Nemotron Bharat” translation plugin for NeMo Data Designer is intended so others can adapt prompts and quality checks to their own sources.

Link to Nemotron 3.5 Super VL

Continued pretraining for Nemotron 3.5 Super VL added 10 billion Indic tokens — 10% of a 100 billion training-token budget — drawn from a twelve-language subset of this corpus plus natural Indic text, the blog says. Stage-level benchmark gains on MILU, Indic MMLU-Pro, Indic ARC and Indic GSM8K are reported in the model’s technical report; the blog stresses those gains reflect a combined synthetic-and-natural mixture, not an isolated test of the synthetic corpus alone.

Releasing soon — not “download today”

The authors repeatedly say the corpus is “releasing soon.” TSN is not treating the full 2.8T-token dataset as already available for immediate download. The blog includes example load_dataset code that asks for a repository ID from the dataset card once the release is up, and advises pinning a revision for reproducibility when it lands.

Scope and limits (authors’ own caveats)

The post is clear that translation broadens access to English source material but does not replace text originally written in Indian languages or guarantee local knowledge and culture. Quality assessment is primarily automated, with sampled human review that varies by language. For the twelve-language workflow, the translator also serves as the judge. The authors recommend combining the corpus with natural Indic text.

Why it matters

Open Indic pretraining data has lagged English-scale corpora for years — a gap AI4Bharat’s Mitesh Khapra has publicly named. A multi-trillion-token synthetic expansion plus a reproducible GPU pipeline is aimed at model builders who need maths, code and technical Q&A in Indian languages, not only web crawl text. Pairing the release with Nemotron 3.5 Super VL’s CPT results gives a concrete “we trained on a slice of this” reference point.

What’s still unknown

• Exact Hugging Face dataset repo ID and licence text at the moment of full release
• When “releasing soon” becomes a dated, pinned dataset revision
• How much of the 2.8T pool other labs will actually mix versus natural Indic data

What to watch

Watch the Hugging Face dataset card and the companion Indic technical report PDF linked from the blog, plus AI4Bharat’s Indic-LLM-Eval-Suite benchmark collection mentioned alongside this release.

What this means

• Companion to Nemotron 3.5 Super VL (22177). Corpus detail expands the CPT story already on TSN.
• ~2.8T tokens, 22 languages, releasing soon. Not framed here as a live download.
• Open pipeline. NeMo Data Designer workflows and prompts are part of the release story.

Sources

1. Hugging Face blog (NVIDIA / AI4Bharat), Will Jennings & Niket Agarwal et al., “Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline”, 16 October 2026
2. TSN, “NVIDIA Releases Nemotron 3.5 Super VL, an Open 120B Multimodal MoE for Agents”

Related stories

• NVIDIA Releases Nemotron 3.5 Super VL, an Open 120B Multimodal MoE for Agents