Param1: a 2.9B‑parameter bilingual language model built from the ground up

India has 22 scheduled languages, many more dialects and more than 1.4 billion people, yet most large language models are trained mainly on English. Param1 is our 2.9 billion parameter bilingual language model, trained from scratch in Hindi and English with India at its core.

The problem: AI's language barrier

Widely used large language models such as GPT-4 and Llama are trained overwhelmingly on English, with Indic languages making up only a tiny share of their data. India accounts for nearly 18% of the world's population.

For Indian users, this gap shows up as:

• Poor comprehension of Indian languages and contexts
• Culturally irrelevant or biased outputs
• Inefficient tokenisation of Indian scripts
• Little support for regional and government use cases

Param1: built for India

Param1 is not a retrofitted Western model. It was trained from scratch as a bilingual Hindi-English model, so it can understand and respond naturally in both languages.

Core design principles

Equitable language representation

Param1 was designed to give at least 25% of its training data to Indic content, far more than mainstream models do. Of the 7.5 trillion tokens it was pretrained on across all three phases, 2.77 trillion are Hindi.

Tokenisation fairness

Alongside Param1, the team built BharatGen-128K, a SentencePiece tokeniser with a 128K vocabulary designed for Indian scripts and word structure. It splits Hindi and other Indian-language text into far fewer tokens than Western-trained tokenisers (see the comparison below). Param1 itself was trained with NVIDIA's Nemotron tokeniser.

Culturally aligned evaluation

The model is evaluated on India-specific benchmarks, including code-mixed reasoning and socio-linguistic robustness, not just English-centric tasks.

Technical architecture

Param1 is a decoder-only causal language model, similar in design to GPT and Llama:

Three‑phase pretraining

Phase 1: Bootstrap training

• 5 trillion tokens (3.48 trillion English and 1.52 trillion Hindi)
• Trained on 64 nodes with 8× NVIDIA H100 GPUs each
• Datasets: FineWeb-Edu, DCLM, Nemotron-CC, Sangraha, Books OCR, Udaan
• Data curation: fastText language detection and quality filtering, toxic content removal, Unicode normalisation, deduplication and PII removal

Phase 2: Factual consistency

• 2 trillion tokens, split evenly between English and Hindi
• Focused on improving knowledge retention and factual correctness
• 20% of the data reused from phase 1, 30% English-Hindi parallel text and 50% new fact-rich monolingual text such as encyclopaedias and textbooks

Phase 3: Long‑context adaptation

• 500 billion tokens (250 billion English and 250 billion Hindi)
• Includes many documents longer than 2,048 tokens, with around 100,000 longer than 8,000 tokens
• Trains long-range dependency handling and coherence across extended inputs

Tokeniser efficiency

The BharatGen-128K tokeniser is more efficient on Hindi than popular alternatives. The table shows fertility, the average number of tokens per word:

Lower scores mean better token efficiency.

Benchmark evaluation and baseline comparisons

To evaluate Param1, we benchmarked its pretrained checkpoint against open-weight language models of similar scale (2–3B parameters). The evaluations cover general reasoning, knowledge recall and, most importantly, Indian language and cultural understanding.

All scores below are zero-shot accuracy on pretrained (PT) checkpoints, so every model is compared under the same setting.

Table 1: Performance on India-centric benchmarks

Table 2: Performance on general language understanding benchmarks. n/a: no comparable score available.

Param1 is competitive with other models of its size. It scores highest on Hindi HellaSwag and TruthfulQA generation, and it outperforms Sarvam-1 on four of the five India-centric tasks. Models trained on much larger and more diverse datasets, such as Qwen2.5, lead on knowledge-heavy benchmarks like MMLU.

Baseline models

We compare Param1 with several widely used models in the 2–3B parameter range:

• Sarvam-1 (2B): A language model trained on Indic languages, offering a direct comparison for Param1's Indian-language capabilities.
• Qwen2.5 (3B): A high-performance model with strong reasoning abilities, trained on a larger and more diverse dataset that includes expert-curated coding and mathematics data. It is a useful baseline for multilingual and logic-based tasks.
• Gemma-2 (2B): The smallest Gemma-2 model from Google DeepMind, pretrained on 2 trillion tokens of mostly English web text, code and mathematics.
• Granite 3.1 (2B dense): IBM's multilingual model trained on 12 trillion tokens, designed for reasoning, instruction following and enterprise use.
• Llama 3.2 (3B): A pruned and distilled model derived from Meta's larger Llama 3.1 models, optimised for general-purpose tasks.

Benchmark suite

Knowledge and reasoning: ARC (Challenge and Easy), MMLU and MMLU-Hi, TriviaQA, LogiQA, TruthfulQA, LAMBADA (standard and OpenAI).

Commonsense and physical reasoning: HellaSwag (English and Hindi), Winogrande, PIQA.

Indian language and cultural proficiency: MILU (Multilingual India-Level Understanding) and SANSKRITI, which tests knowledge of Indian culture with over 21,000 multiple-choice questions on history, festivals, cuisine and more.

Alignment and safety

Prometheus evaluation

To measure generation quality, we used the Prometheus-Eval framework with GPT-3.5-Turbo as the judge. It scores open-ended responses on grammar, input-output alignment and semantics, giving a scalable quality score without human annotation.

In Hindi, Param1 scored 3.876 overall, ahead of Sarvam-1 (3.057), Gemma-2 (2.372) and Llama 3.2 (2.165). In English, it scored 3.259, close to Sarvam-1 (3.318) and ahead of Gemma-2 (2.867).

Toxicity evaluation

We evaluated safety using the Toxigen benchmark through LLM360's Safety360 suite. Param1 was prompted with neutral and adversarial templates designed to draw out toxic or stereotyped content, and its outputs were scored with the suite's RoBERTa-based Toxigen classifier for overall toxicity and identity-based abuse.

Param1 shows lower or comparable toxicity rates relative to Sarvam-1 and other multilingual baselines.

Instruction fine‑tuning

The instruction-tuned release, Param-1-2.9B-Instruct, was trained on top of the pretrained checkpoint:

• About 0.8 million bilingual instruction-response samples
• Public and in-house sources, filtered for quality, safety and cultural relevance
• Domain content from Indian governance, education, legal, agriculture and public services
• Three epochs on 32 NVIDIA H200 GPUs

Real‑world use cases

Param1 is intended as a foundation that developers and researchers can adapt for sectors such as:

• Governance: multilingual digital services
• Education: AI tutors for diverse learners
• Healthcare: region-specific medical assistants
• Legal: understanding local laws and regulations
• Agriculture: farmer outreach in regional languages

Infrastructure and scale

Param1 was trained on Yotta's SLURM-managed GPU cluster:

• 8× NVIDIA H100 GPUs per node
• High-speed InfiniBand interconnect
• NVIDIA NeMo framework for distributed training

Why Param1 matters

Param1 shows that linguistic inclusion can be built into a model from the first stage of pretraining, rather than added afterwards through fine-tuning. It also shows that this can be done without giving up general-purpose performance.

Get started with Param1

The pretrained Param1 checkpoint is available on AIKosh, and the instruction-tuned Param-1-2.9B-Instruct is on Hugging Face, both as open-weight models for public use and research. The full technical report is on arXiv.

Param1 has since been followed by Param2, our next generation of open-weight text models. See all our text models.