BhashaBench‑Krishi (BBK): measuring AI's gap in Indian agricultural knowledge

The gap in agricultural AI

India's agriculture supports around 600 million farmers and feeds 1.4 billion people. AI promises to help through precision agriculture, crop monitoring and advisory systems, but current AI models still fall short on the domain-specific knowledge that Indian agriculture demands.

The problem goes beyond translation or language barriers. Many AI systems cannot reliably distinguish rabi from kharif crops, do not understand soil health parameters specific to Indian conditions, and miss the relationship between monsoon patterns and farming practices that have evolved over centuries.

BhashaBench-Krishi is a benchmark grounded in government agricultural exams and built specifically for Indian agricultural knowledge. It is one of the four benchmarks in BhashaBench V1. The results show significant performance gaps, and those gaps will shape whether digital tools help farmers or leave many of them behind.

Why current AI benchmarks miss the mark

Existing agricultural benchmarks

A number of specialised agricultural AI benchmarks have appeared in recent years:

• AgriBench/MM-LUCAS: EU field image classification (2024)
• BVL QA Corpus: German agricultural QA (2024)
• AgXQA: irrigation-focused extractive QA (2024)
• CROP Benchmark: rice and corn MCQs (2024)
• AgEval: stress identification and quantification (2024)
• AgroEvals with AgroGPT: expert-tuned dialogues (2024)
• SeedBench: breeding and gene-trait reasoning (2025)
• CDDM and CDwPK-VQA: multimodal crop disease diagnosis (2024–2025)
• WheatRustVQA: rust severity detection (2024)
• Krishiq-BERT: Kannada agricultural QA (2024)
• Embrapa Dairy QA: Portuguese dairy queries (2022)
• Agronomy MCQ 98: general MCQs (2024)

None of them fully captures the context-rich, region-specific knowledge required for real-world farming in India. Each has clear limitations in the Indian context:

• Geographic mismatch: many focus on EU, Chinese or South American crops and practices, with little relevance to Indian agro-ecological zones.
• Narrow crop scope: they are often restricted to a handful of globally important crops (rice, wheat, maize) rather than India's 50+ major crops and thousands of local varieties.
• Task isolation: they are strong on image recognition, MCQ answering or disease classification in isolation, but rarely bring in policy knowledge, scheme eligibility or seasonal agronomy.
• Language limitations: most are in English or a single non-Indian language, with no bilingual coverage of English and Indian languages.
• Exam disconnect: few, if any, are grounded in official government agricultural examination questions, which are the closest standardised proxy for real-world farm advisory competence.

Broader benchmarks such as MMLU, MILU and Sanskriti share the same flaw: where they include agricultural questions, those questions focus on generic scientific principles rather than the practical decisions farmers face.

An example

A traditional benchmark question tests static, textbook recall:

"What is the chemical formula for photosynthesis?"

A BhashaBench-Krishi question, posed in Hindi, tests contextual understanding:

"निम्नलिखित में से कौन-सा कथन मृदा जल विभव और दोमट, बलुई तथा चिकनी मिट्टी के नमी अंश के बीच सही संबंध को स्पष्ट करता है?"

(Which statement correctly explains the relationship between soil water potential and moisture content in loamy, sandy and clay soils?)

1. दिए गए विभव पर दोमट मृदाओं की अपेक्षा चिकनी मिट्टियाँ अधिक जल रोकती हैं।
2. दिए गए विभव पर चिकनी मिट्टियों की अपेक्षा दोमट मृदाएँ अधिक जल रोकती हैं।
3. दिए गए विभव पर बलुई मृदाओं की अपेक्षा दोमट मृदाएँ अधिक जल रोकती हैं।
4. दिए गए विभव पर दोमट मृदाओं की अपेक्षा चिकनी मिट्टियाँ कम जल रोकती हैं।

The first question tests a memorised fact that does not change with context. The second:

• requires an understanding of soil physics and water retention across loamy, sandy and clay soils
• tests applied knowledge needed for irrigation, crop management and soil conservation
• reflects real farming decisions that affect yield and sustainability
• is asked in Hindi, as it would be for many Indian farmers and agronomists

The scale of India's agricultural challenge

Indian agriculture brings complexities that generic or translated datasets cannot cover:

• 15 major agro-climatic zones, each with its own crop patterns
• over 50 major crops, with region-specific varieties and practices
• three cropping seasons: kharif, rabi and zaid
• state-specific policies that shape crop planning and subsidies
• a mix of indigenous and modern farming techniques
• hundreds of government schemes with varying eligibility
• market factors such as MSP, procurement policies and export restrictions

This calls for exam-validated benchmarks that measure whether a model can give actionable, local and policy-aware advice, not just recite general agronomy facts.

About BhashaBench‑Krishi

BhashaBench-Krishi is part of the BhashaBench series and is, to our knowledge, the largest agricultural knowledge benchmark for Indian languages. It currently supports English and Hindi, with more Indian languages planned for later versions.

Exam spectrum: from field to research lab

The 55+ exams span the full range of agricultural roles:

• Rural outreach and extension: NABARD Grade A, MP RAEO and NSCL recruitment, focused on rural development, community engagement and farmer advisory work.
• Applied agriculture and agribusiness: NFL Management Trainee and IBPS-AFO, linking farming, markets and industry.
• Postgraduate entry: ICAR AIEEA PG, with wide coverage of MSc-level disciplines.
• Advanced research: ICAR AICE PhD, covering areas such as Genetics, Plant Pathology and Seed Science.
• University and fellowship programmes: JRF fellowships and university PG/PhD entrance exams in Agronomy, Horticulture and Biotechnology.
• National recruitment and scientist roles: ASRB scientist recruitment and Pre-PG tests.

Subject domains

More than 25 domains and 270 topics cover the agricultural and allied sciences:

• Core agricultural sciences: Agronomy, Soil Science, Crop Sciences, Plant Pathology, Entomology, Nematology, Genetics and Plant Breeding.
• Advanced and applied fields: Agricultural Biotechnology, Seed Science and Technology, Plant Physiology, Agricultural Engineering and Technology (mechanisation, irrigation, remote sensing).
• Allied life sciences: Animal Sciences, Veterinary Sciences, Dairy and Poultry Science, Fisheries and Aquaculture.
• Supporting disciplines: Agricultural Microbiology, Chemistry and Biochemistry, Agricultural Extension Education.
• Economic and digital agriculture: Agricultural Economics and Policy, Agriculture Information Technology.
• Skills and awareness: General Knowledge and Reasoning, Agriculture Communication, Agri-Environmental and Allied Disciplines (sustainability, climate-smart practices).

Difficulty levels

• Easy (6,754 questions): basic agricultural knowledge and facts.
• Medium (6,941 questions): the largest group, requiring interpretation and application of concepts in practical contexts.
• Hard (1,710 questions): advanced reasoning, multi-step problem solving or combining knowledge across areas.

Question types

• Multiple choice (13,550): knowledge recall and application.
• Assertion-reasoning (648): judging statements and the logical link between them.
• Match the column (949): pairing related items.
• Rearrange the sequence (209): ordering steps or processes, which matters for procedural knowledge in agriculture.
• Fill in the blanks (49): recall of specific terms or values.

Methodology: from government exams to AI benchmark

Stage 1: Data collection

We sourced exam content from trusted government and institutional sources:

• Official exam portals: including ICAR, NABARD Grade A, MP RAEO, NSC and state agricultural universities.
• Government publications: agricultural department question banks and official releases.
• Institutional sources: agricultural colleges and research institutes.
• Verification: cross-referencing questions across multiple official sources, with domain expert validation.

Stage 2: OCR processing

• OCR model: Surya, chosen for its recognition of Indian languages and technical terms.
• Multi-script handling: English, Hindi and agricultural terminology.
• Format preservation: keeping question structure, numbering and formatting intact.

Stage 3: Model‑assisted post‑correction

• Correction model: Qwen3-235B, selected for its multilingual and domain understanding.
• Context-aware correction: accurate handling of agricultural jargon and technical content.
• Quality validation: several correction passes combined with domain-specific consistency checks.

Stage 4: Topic classification

• Source-based tagging: questions were first tagged with the exact subjects and sections from the official exam papers, so labels match the curriculum.
• LLM-guided consolidation: Qwen3-235B grouped related exam topics into broader domains. For example, "Crop Physiology", "Crop Nutrition" and "Crop Protection" were consolidated under "Crop Sciences".
• Two-level hierarchy: detailed exam sections map to 25+ higher-level domains, supporting both granular and summary analysis.
• Accuracy: the model assigned domains correctly 93% of the time on a manually verified validation set. Domain experts then reviewed and refined the classifications.

Stage 5: Linguistic and cultural validation

• Expert team: 30 agricultural language specialists reviewed questions for clarity, correct terminology and cultural sensitivity.
• Checks: grammar, translation accuracy and fit with regional farming contexts across agro-climatic zones.
• Outcome: about 85% of questions passed this initial review; the remaining 15% were corrected by the language experts.

Stage 6: Domain specialist evaluation

• Specialists: about 10 experienced agricultural scientists and practitioners assessed questions for scientific accuracy and real-world relevance.
• Outcome: roughly 96% of questions were confirmed as technically valid and relevant.

Results: AI's agricultural knowledge gap

Overall model performance

We evaluated 29+ language models on BhashaBench-Krishi. Performance varies widely:

• Higher-performing models such as GPT-4o, Qwen3-235B-A22B-Instruct-2507 and gpt-oss-120b score above 70% on the overall benchmark and on the English subset.
• On the Hindi subset, scores are generally lower, with top performers at around 60–65%.
• Lower-performing models cluster in the 25–40% range, showing limited domain knowledge or language adaptation for agriculture.

These results show a clear gap in agricultural expertise, especially in multilingual settings, with room for improvement across most models.

Performance by domain

Highest-performing domains. Agricultural Biotechnology, Plant Sciences and Physiology, and Veterinary Sciences show consistently strong accuracy, often above 80% for top models. These fields benefit from well-established scientific knowledge and standardised terminology. Agricultural Chemistry and Biochemistry, and Agronomy, also score well.

Moderate domains. Soil Science, Agricultural Engineering and Technology, and Animal Sciences fall in the 40–70% range. These fields combine universal scientific principles with regional, practical knowledge. Agricultural Extension Education and Agriculture Communication also score in the mid-range, which points to difficulty with culturally nuanced, communication-heavy content.

Lowest-performing domains. Agri-Environmental and Allied Disciplines, Nematology and Regional Crop Management score lower, often below 40–50%, reflecting the location-specific expertise they require. Rural Development and Policy Understanding are also difficult, since they need contextual and social knowledge beyond textbook facts.

Performance by difficulty

• Easy questions get the highest accuracy across the board. GPT-4o and Qwen3-235B reach nearly 79% and 78%, and most models do much better here than on harder questions.
• Medium questions are a moderate challenge, with accuracy generally 10–20 percentage points lower than on easy questions. Top models stay in the 65–70% range.
• Hard questions get the lowest scores, often below 40% even for the best models.

Performance by question type

• Multiple choice: the highest accuracy overall. Top models consistently score above 70%.
• Rearrange the sequence: tests logical ordering and multi-step reasoning. Top models reach around 73–75%, slightly below MCQ.
• Match the column: tests associations between concepts. Leading models generally score 60–68%.
• Assertion-reasoning: needs deeper inference. Top models reach 70%, but many fall well behind.
• Fill in the blanks: the hardest format, with accuracy often below 60%, showing difficulty in producing a precise word or phrase without options.

Why this matters

Agricultural AI applications at risk

• Precision agriculture: current models struggle to interpret soil test results in Indian contexts. Fertiliser recommendations lack an understanding of local soil conditions, and crop monitoring tools often miss regional varieties and practices.
• Advisory chatbots: these often give generic or inaccurate guidance on seasonal activities, fall short on government schemes and subsidy eligibility, and do not adapt advice to specific agro-climatic zones.
• Smart farming tools: irrigation recommendations overlook local water management practices, pest advice is disconnected from regional pest biology, and price predictions ignore local market dynamics.

The economic stakes

• India's ₹35 lakh crore (about US$400 billion at ₹88 = US$1) agricultural sector is exposed to poorly performing AI.
• 600 million farmers risk being left out of AI tools that could raise productivity and incomes.
• Food security for 1.4 billion people depends on efficient agricultural systems.
• The digital divide between technology-enabled large farms and smallholders could widen.

Beyond agriculture

The same gaps appear in other domain-specific AI applications. Medical AI misses regional disease patterns, legal AI lacks jurisdiction-specific insight, and educational AI may not know local curricula. Traditional knowledge, regional practices and local institutions are poorly represented, and technical terms in Indian languages, or concepts with no direct translation, remain hard for AI systems.

Future directions

Immediate research priorities

Data collection:

• Add state-level agricultural competitive exams.
• Expand practical, field-based problem scenarios.
• Document indigenous and traditional agricultural knowledge.
• Keep current schemes, policies and technologies up to date.

Model development:

• Pre-train domain-specific agricultural language models.
• Build multilingual agricultural AI beyond Hindi and English.
• Develop multimodal systems that combine images, text and sensor data.
• Create models that keep their agricultural knowledge current.

Long‑term goals

We want AI systems that understand the full agricultural ecosystem, combine traditional knowledge with modern science, adapt to changing conditions, and give location-specific guidance. These tools should be accessible to smallholder farmers, support all major Indian languages, be culturally sensitive in their recommendations, and be affordable for different kinds of farms.

Conclusion

BhashaBench-Krishi shows that current language models, for all their capabilities:

• have large blind spots in specialised agricultural knowledge
• do not gain professional competence from scale alone
• struggle with the cultural and regional context that real farming depends on
• are less ready for agriculture than is commonly assumed

Building AI that serves agriculture needs domain-first development, close work with farmers, agricultural scientists and extension workers, evaluation on benchmarks that test practical competence, and long-term investment in research.

The full BhashaBench-Krishi dataset is available on Hugging Face and integrated with lm-eval-harness for ongoing research. We hope it helps make sure AI in agriculture leaves no farmer behind.

Access the benchmark on Hugging Face: bharatgenai/BhashaBench-Krishi

For questions or feedback, please get in touch.