Eventually, the Steam Drill Always Wins: "Law Professors Prefer AI Over Peer Answers"

Key Takeaways

• LLMs won 75% of blind comparisons against human professors
• Gemini 2.5 Pro and NotebookLM outperformed most instructors
• AI harmful rates stayed below 4%, lower than many professors
• Preference persisted across recall, hypothetical, and policy questions
• Stylistic features explained only part of AI's advantage

Pulse Analysis

The Stanford experiment provides rare, data‑driven evidence that large language models can rival seasoned law professors in delivering concise, accurate contract‑law explanations. By using a forced‑choice, blinded methodology, the researchers eliminated bias and revealed that both Gemini 2.5 Pro and the retrieval‑augmented NotebookLM consistently outperformed the majority of human instructors. This performance held steady across question types—from straightforward doctrinal recall to complex hypotheticals and policy debates—suggesting that AI’s strength is not limited to rote memorization but extends to nuanced legal reasoning.

For law schools, the implications are twofold. First, AI tutors could alleviate faculty workload, offering reliable, on‑demand support during office hours and freeing professors to focus on higher‑order pedagogy and research. Second, the low harmfulness rates—under 4%—challenge the prevailing narrative that AI-generated legal advice is inherently risky. However, the study also warns that stylistic cues such as length or clarity only partially account for the preference, indicating that LLMs may be capturing deeper, discipline‑specific standards that educators value.

Adoption will hinge on policy and ethical frameworks. Institutions must develop guidelines to ensure transparency, prevent over‑reliance, and address potential biases in model training data. As AI continues to infiltrate professional education, the legal academy faces a pivotal choice: integrate these tools as complementary resources or risk lagging behind peers who already leverage AI to enhance learning outcomes. The data suggests that, when responsibly deployed, generative AI could become a permanent fixture in the law classroom.