Standardizing Gen Al Service Evaluation, An API-Centric Benchmarking Approach with David Kanter

Why It Matters

Standardizing AI inference measurement through an API‑first, community‑driven benchmark gives buyers reliable performance data and accelerates adoption of emerging generative models.

Key Takeaways

• MLPerf moving from spreadsheet to modern API-driven benchmark platform.
• New decoupled architecture enables minutes‑fast system setup and testing.
• API‑centric approach mirrors generative AI deployment via OpenAI‑style endpoints.
• Benchmarks now capture concurrency, latency, and throughput across utilization curves.
• Community‑driven, inclusive model supports custom data sets and reproducibility.

Summary

The presentation announced a major overhaul of MLPerf’s inference benchmark, shifting from a legacy spreadsheet‑based, C++ load‑generator model to a modern, API‑centric framework that mirrors how generative AI is delivered today. By adopting a decoupled architecture that communicates with systems under test via standard OpenAI‑style endpoints, MLPerf aims to make benchmark deployment a matter of minutes rather than weeks.

Key insights include a focus on relevance, fairness, reproducibility and inclusiveness, reinforced by partnerships with ISO and other standards bodies. The new platform supports rapid integration of new data sets, custom workloads, and fine‑grained concurrency testing that captures throughput, latency, and token‑per‑user metrics across utilization curves. This approach also enables step‑function visualizations that avoid misleading interpolations, preserving trust in reported performance.

Examples highlighted the benchmark’s real‑world impact: a national lab used MLPerf scores to justify a multi‑petaflop supercomputer purchase, and the community has observed roughly 50× performance gains over the past three to four years. The API‑driven design now lets users plug in proprietary data, run on‑prem or cloud services, and generate reproducible, auditable results in minutes.

The implications are significant for enterprises and vendors alike. Faster, more realistic benchmarking lowers the barrier for AI procurement, supports clearer service‑level agreements, and provides a neutral yardstick as generative AI models evolve at a bi‑weekly cadence. Ultimately, the shift positions MLPerf as the de‑facto standard for measuring AI inference performance in a rapidly changing market.

Original Description

David Kanter detailed the ongoing evolution of MLPerf benchmarks, which have been an industry standard for seven years. He highlighted the need for fundamental changes, particularly in the visualization of results, moving from an outdated, spreadsheet-like format to a more modern and understandable interface. MLPerf, backed by MLCommons, is widely used by over 100 members for internal testing, showcasing capabilities, and informing purchasing decisions. Its success stems from core principles of relevance, fairness, neutrality, reproducibility, and inclusiveness, all working together to foster trust and drive industry advancement.

The landscape of AI performance has radically shifted with the explosion of generative AI, marked by immense user adoption and an unprecedented velocity of change, with new models appearing almost fortnightly. To keep pace and better serve buyers, MLPerf is transitioning to an API-centric benchmarking approach. This involves moving away from a complex, locally installed load generator to a decoupled, Python-based test infrastructure that interacts with the system under test via a standard API, similar to the OpenAI API. This new architecture simplifies setup, accelerates the integration of new datasets and benchmarks, and supports comprehensive measurement across varying concurrency levels, capturing critical metrics like time-to-first token, throughput, and full response latency without relying on interpolation.

This strategic shift aims to significantly increase the velocity of benchmark submissions, allowing for more frequent updates than the current six-month cycle, while rigorously maintaining peer review and auditability to preserve trust. Kanter acknowledged the complex and multidimensional challenge of assessing quality in generative AI and agentic applications, a problem MLPerf is actively addressing in its long-term roadmap. He concluded by inviting feedback from the community, especially from enterprise buyers and analysts, to ensure the benchmarks remain relevant, understandable, and valuable for the widespread deployment of generative AI.

Presented by David Kanter, Co-Founder and Head of MLPerf, MLCommons. Recorded live in San Jose, California, on May 14, 2026 as part of AI Field Day 8. Watch all three Community Presentations at https://techfieldday.com/appearance/ai-field-day-8-community-presentations/ or visit https://TechFieldDay.com/event/aifd8/ to learn more.