Evaluating Different AI's on African Livestck Knowledge

Key Takeaways

• Llama 3.1 8B scored 43% on Nigerian veterinary benchmark
• Benchmark contains 420 questions across six livestock categories
• Standard AI tests overlook critical regional knowledge gaps
• Future study will compare Claude, GPT‑4o, Gemini models

Pulse Analysis

Artificial intelligence is rapidly entering the agricultural sector, promising to boost productivity and disease management for smallholder farms. Yet most performance metrics rely on datasets rooted in Western contexts, leaving a blind spot for regions like sub‑Saharan Africa where livestock practices are shaped by centuries of ethnoveterinary knowledge. By constructing a 420‑question test drawn from Nigerian veterinary curricula, published ethnoveterinary literature, and field expertise, the researcher highlights the urgent need for benchmarks that reflect local realities, ensuring AI tools are trustworthy where they are most needed.

The initial evaluation of Meta’s Llama 3.1 8B, accessed via Groq, revealed a modest 43% accuracy across six categories, from breed characteristics to disease identification. This result underscores that a model can excel on generic benchmarks while faltering on niche, high‑stakes domains. The scoring rubric (0‑1‑2) captures partial knowledge, offering a nuanced view of model competence. Such findings are especially relevant as AI‑driven advisory platforms are already being piloted in African agricultural extension services, where erroneous guidance could exacerbate animal loss and economic hardship.

Looking ahead, the project will broaden its scope to include Claude Sonnet, GPT‑4o, and Gemini 1.5 Pro, providing a comparative landscape of leading large‑language models on African livestock knowledge. The forthcoming paper aims to set a precedent for domain‑specific AI safety evaluations, encouraging researchers, policymakers, and investors to allocate resources toward culturally aware benchmark development. By filling this evaluation gap, the AI community can better safeguard the health of livestock and the livelihoods of farmers across the continent.