GPT-4 can write poetry and solve complex maths problems, so why does it struggle with basic questions about Indian agriculture or Ayurvedic principles? Most AI benchmarks test Western knowledge systems, which leaves a large blind spot for India-specific domains.
BhashaBench V1 is India's first comprehensive, bilingual benchmark designed to evaluate how well AI models understand Indian knowledge systems.
Why this matters
Imagine a farmer in Punjab asking an AI assistant about pest management for their wheat crop, or a law student in Patna preparing for the civil services exam. Current AI models might give generic answers, but do they understand Indian agriculture, the complexities of Indian law, or the depth of Ayurveda?
India is not just another market for AI. It has:
- 93 million agricultural households who need advice on region-specific crops and practices
- millions of legal professionals and citizens navigating one of the world's most complex legal systems
- a traditional medicine system that millions rely on for healthcare
- a fast-changing financial landscape with its own regulatory frameworks
When we tested 29 leading language models on Indian knowledge, the results were sobering. GPT-4o scored 76% on legal questions but just 59.74% on Ayurveda. Models consistently performed worse on Hindi content than on English, and smaller models struggled even more. AI models are not yet ready for India's knowledge landscape.
Four benchmarks in one suite
BhashaBench V1 is not a single test. It is a suite of four domain-specific benchmarks, each designed to evaluate AI on an important Indian knowledge system. Each has its own detailed write-up, linked below.
BhashaBench‑Krishi (BBK): agriculture
India's first large-scale agricultural AI benchmark, built from 55+ government agricultural exams across the country.
- 15,405 questions spanning 25+ agricultural domains and 270+ topics
- coverage from soil science to agricultural policy, and agronomy to horticulture
- questions in English (12,648) and Hindi (2,757)
- real exam questions that test practical, region-aware farming knowledge
Why it matters: when a farmer asks about pest management for a Basmati crop or the best sowing time for cotton in Maharashtra, the model needs to understand India's agro-ecological zones, not just generic farming principles.
Read more about BhashaBench-Krishi
BhashaBench‑Legal (BBL): Indian law
The first large-scale legal knowledge benchmark built for Indian jurisdictions, based on 50+ official law exams.
- 24,365 questions spanning 20+ legal disciplines and 200+ topics
- coverage from constitutional law to cyber law, and civil litigation to intellectual property
- questions in English (17,047) and Hindi (7,318)
- practical, jurisdiction-specific legal knowledge
Why it matters: Indian law is deeply contextual. Understanding the criminal codes (the IPC and now the Bharatiya Nyaya Sanhita), the CPC, the Constitution and state-specific regulations takes specialised knowledge that generic legal training does not capture.
Read more about BhashaBench-Legal
BhashaBench‑Finance (BBF): financial systems
India's first comprehensive financial knowledge benchmark, drawn from 25+ government and institutional financial exams.
- 19,433 questions across 30+ financial domains
- coverage from banking services to taxation, and corporate finance to rural economics
- questions in English (13,451) and Hindi (5,982)
- India-specific regulations, policies and financial practices
Why it matters: India's financial ecosystem is distinctive, from UPI processing billions of transactions a month to its own taxation frameworks. Models need to understand India's regulatory landscape, not just global finance theory.
Read more about BhashaBench-Finance
BhashaBench‑Ayur (BBA): traditional medicine
The first comprehensive benchmark of Ayurvedic knowledge, grounded in classical texts and modern Ayurvedic education.
- 14,963 questions across 15+ Ayurvedic disciplines
- coverage from Kayachikitsa (internal medicine) to Panchakarma, and Dravyaguna to diagnostics
- questions in English (9,348) and Hindi (5,615)
- links classical texts with contemporary Ayurvedic practice
Why it matters: in India, Ayurveda is not alternative medicine. It is a complete healthcare system with rigorous educational standards, and models need to understand it with the same depth as modern medicine.
Read more about BhashaBench-Ayur
What makes BhashaBench different
- Authentic sources: every question comes from real government exams, professional certifications and institutional assessments, the same tests millions of Indians take for education and careers. Subject matter experts validated them.
- Granular evaluation: with 90+ subdomains and 500+ topics, we do not just report that a model scored 65% overall. We show that it does well on international finance but struggles with seed science, or handles constitutional law but fails at cyber law.
- Bilingual: 74,166 questions in total, 52,494 in English and 21,672 in Hindi, reflecting how India actually communicates. This is not just translation; questions keep their cultural context in both languages.
- Diverse task types: multiple choice, assertion-reasoning, fill in the blanks, match the column, reading comprehension and more, each testing a different dimension of understanding.
- Difficulty levels: questions are graded Easy, Medium and Hard, showing where models succeed and where they reach their limits.
What we found
We evaluated 29 language models across BhashaBench V1. The results show critical gaps.
- Performance varies widely across domains. Models that do well in one domain often struggle in others. Cyber Law and International Finance are relatively strong; Panchakarma, Seed Science and Human Rights remain weak.
- The Hindi gap is real. In every domain, models performed better on English content than on Hindi, even when testing the same knowledge. This reflects the resource gap between English and Indian languages.
- Domain depth matters more than model size. Larger models generally performed better, but the gaps narrowed in specialised contexts. Depth in a domain matters more than breadth of training.
- Even the best models have room to grow. Top models scored between 34% and 76% across domains. BhashaBench is far from solved, even for state-of-the-art models.
Real‑world impact
BhashaBench is about building AI that serves India's needs:
- Agriculture: wider access to expert crop advisory, pest management and sustainable farming practices for farmers in remote areas who cannot easily reach agricultural universities.
- Legal: better access to legal information and basic legal literacy in a country where justice remains out of reach for millions because of complexity and cost.
- Finance: stronger financial literacy and support for India's digital payments ecosystem, helping millions with banking, taxation and financial planning.
- Healthcare: preserving and sharing traditional medical knowledge, supporting Ayurvedic practitioners and patients.
Built for the community
BhashaBench is released under the CC BY 4.0 licence. We are releasing:
- complete datasets for all four benchmarks
- evaluation code using lm-eval-harness
- model performance leaderboards
- documentation and resources
We are releasing it openly because solving India's AI challenges needs researchers, developers, organisations and institutions working together.
What's next
BhashaBench V1 is a first step. We are working on:
- More languages: expanding beyond English and Hindi to Tamil, Telugu, Bengali and other Indian languages.
- More domains: traditional crafts, regional governance and indigenous knowledge systems.
- More question types: long-form reasoning, code generation and multimodal tasks.
Get started
BhashaBench is built for anyone who is:
- developing LLMs and needs rigorous India-centric evaluation
- deploying AI in India and wants to benchmark performance
- researching multilingual AI with cultural context
- building applications for agriculture, legal tech, fintech or healthcare in India
- Access the datasets on Hugging Face: Krishi, Legal, Finance and Ayur
- Read the BhashaBench V1 paper on arXiv
- View the evaluation code on GitHub
- See the leaderboard and submit your model results
Have feedback, or want to help expand BhashaBench to more languages and domains? Get in touch.
BhashaBench is a collaborative effort to make sure AI development serves India's billion-plus people with the domain expertise and cultural context they deserve.
