BhashaBench‑Ayur: evaluating AI's knowledge of Ayurveda

Ayurveda is one of the oldest systems of medicine in the world, with roots going back thousands of years, and it still serves millions of people through its approach to health, disease prevention and treatment. As AI enters healthcare, a basic question follows: can today's AI systems understand and apply this body of knowledge?

BhashaBench-Ayur is India's first comprehensive benchmark for evaluating AI models on Ayurvedic knowledge. It draws on government examinations and institutional assessments from across the country, and it is one of the four benchmarks in BhashaBench V1.

The evaluation shows some promising capabilities and some significant gaps. Both matter for building AI that can serve India's traditional healthcare sector without flattening the depth of Ayurvedic knowledge.

Why Ayurvedic AI evaluation matters

Beyond translation

Ayurveda challenges AI in ways that go well beyond translating terms or looking up medical vocabulary:

• Philosophical foundations: core concepts such as Tridosha theory (Vata, Pitta, Kapha), Panchamahabhuta (the five elements) and Prakriti-Vikriti analysis need conceptual understanding, not memorisation.
• Integrative reasoning: diagnosis and treatment combine constitutional analysis, seasonal factors, lifestyle and individual patient characteristics.
• Classical texts: references from the Charaka Samhita, Sushruta Samhita and Ashtanga Hridaya need context built up over centuries of scholarly commentary.
• Cultural context: Ayurvedic practice is embedded in Indian cultural, spiritual and social life, which shapes treatment and patient interaction.
• Practical application: moving from theory to clinical decisions means reasoning about many interconnected health factors.

Integration with modern healthcare

As India strengthens its AYUSH sector and promotes integrative medicine, AI systems that understand Ayurveda become more important:

• Clinical decision support: tools for Ayurvedic practitioners must understand traditional diagnosis, therapeutic principles and treatment protocols.
• Patient education: digital health platforms need to explain Ayurvedic concepts accurately without oversimplifying them.
• Research: AI supporting Ayurvedic research must connect traditional knowledge with modern scientific methods.
• Quality assurance: systems that evaluate Ayurvedic education and practice need a deep understanding of its standards.

About BhashaBench‑Ayur

BhashaBench-Ayur is, to our knowledge, the largest benchmark for evaluating AI on traditional medicine. It is built from sources across India's Ayurvedic education and certification system.

Domains covered

The benchmark brings together content from more than 50 government examinations and institutional assessments, covering every major branch of Ayurvedic knowledge and practice.

Difficulty and question types

Difficulty levels:

• Easy (53%, 7,944 questions): fundamental concepts and terminology
• Medium (42%, 6,314 questions): applied knowledge and clinical reasoning
• Hard (5%, 705 questions): complex analysis and advanced practice

Question types:

• Multiple choice (98.4%, 14,717 questions): the standard format
• Fill in the blanks (1.2%, 178 questions): precise terminology
• Match the column (0.3%, 41 questions): relationships between concepts
• Assertion-reasoning (0.2%, 27 questions): logical analysis

Results

We evaluated 29 language models. The results show where AI understands traditional medicine and where it does not.

Overall performance

• Top models: Qwen3-235B (58.2% overall) and DeepSeek-v3 (57.1% overall) score highest.
• English: leading models reach 60.25% on English Ayurvedic content.
• Hindi: top models reach 54.78%. The 5–6% drop reflects how hard these concepts are in Indian languages, and shows clear room for improvement.

Across models, scores fall into rough tiers:

• High performers (55%+): Qwen3-235B (58.2%) and DeepSeek-v3 (57.1%).
• Mid-tier (40–55%): Gemma-2-27B (52.3%) and the Llama-3.1-8B series.
• Domain-specific models: AyurParam (39.97%) is competitive despite its smaller parameter count.
• Lower tier (below 35%): various smaller and base models.

Performance by domain

Models do best in research, public health and diagnostics, and worst in the therapeutic and pharmacological core of Ayurveda: Panchakarma and Dravyaguna both score below 50%.

Performance by difficulty

• Easy: the top score is 65.18%. Most models fall in the 40–60% range, so fundamental principles are reasonably well captured in training data.
• Medium: the top score is 50.74%. Models have moderate success with applied scenarios but struggle with multi-step reasoning.
• Hard: the top score is 46.24%, showing the limits of current models on advanced concepts.

Performance by question type

• Fill in the blanks (62.96%): models handle precise Ayurvedic vocabulary and term identification well.
• Assertion-reasoning (59.26%): moderate success with cause-and-effect reasoning and Ayurvedic logic.
• Match the column (58.34%): moderate success connecting and classifying concepts.
• Multiple choice (51.69%): the baseline on the main format, which makes up most of the benchmark.

Key insights

Strengths: models do well on research-related questions, well-structured domains, preventive medicine and lifestyle, and reasonably well on policy and administration.

Limitations: models struggle with complex treatments, herbal pharmacology and classical formulations, culturally embedded practices and advanced Ayurvedic reasoning.

Implications: there is a clear need for domain-specific model development, for close involvement of Ayurvedic practitioners, and for a phased approach to using AI in traditional medicine.

Real‑world implications

Traditional healthcare

• Clinical decision support: gaps in understanding complex procedures could lead to inappropriate treatment recommendations or missed therapeutic options.
• Digital health platforms: AI may give inaccurate guidance on formulations, Panchakarma protocols or constitutional analysis.
• Education: AI tutors might explain classical concepts, traditional diagnostic methods or therapeutic principles poorly.
• Telemedicine: remote consultation tools may miss the assessment approaches that Ayurvedic practice relies on.

Societal and cultural impact

• Healthcare access: AI that does not understand Ayurveda could create barriers for the millions who rely on it as primary or complementary care.
• Preservation: poor AI representation of traditional knowledge could contribute to its erosion.
• Integrative medicine: limited understanding may hold back efforts to combine traditional and modern approaches.
• Global reach: weak AI comprehension could limit how well India shares traditional medicine knowledge with the world.

Sector by sector

• AYUSH sector: needs specialised AI to support education, research and practice.
• Pharmaceuticals: needs AI that understands traditional formulations, herbal interactions and classical preparation methods for development and safety assessment.
• Research: needs tools that connect traditional knowledge with modern methods.
• Policy: needs AI that understands traditional medicine frameworks for evidence-based planning.

Future directions

Training data:

• classical Sanskrit texts with modern translations and commentaries
• clinical case studies from practitioners across India
• documentation of regional variations in practice
• research connecting traditional principles with modern scientific validation

Model development:

• pre-training on a traditional medicine corpus that keeps its cultural context
• multilingual capability, including Sanskrit terminology and classical references
• combining traditional reasoning patterns with modern analysis
• keeping models current as practice develops

In the long term, we want AI that understands traditional Indian medicine from classical texts to contemporary practice, gives culturally appropriate and therapeutically sound guidance, and serves both practitioners and patients seeking integrative care.

Conclusion

BhashaBench-Ayur shows both promise and substantial challenges. Current models perform reasonably in systematic and research-oriented domains, but significant gaps remain in the therapeutic knowledge at the heart of Ayurvedic practice.

• Knowledge complexity: Ayurveda needs more than factual recall. It needs an understanding of interconnected principles, cultural context and reasoning developed over centuries.
• Cultural authenticity: AI for traditional medicine must respect the cultural and philosophical foundations of the practice.
• Collaboration: combining traditional knowledge with AI needs technology developers, practitioners and cultural experts working together.
• Specialised development: general-purpose models, whatever their size, cannot adequately serve traditional medicine without targeted development.

The benchmark is available on Hugging Face so researchers and practitioners can build on it, and help make sure AI supports the healing traditions that millions still rely on.

Access the benchmark on Hugging Face: bharatgenai/BhashaBench-Ayur