Agent Skills Look Great in Benchmarks but Fall Apart Under Realistic Conditions, Researchers Find

Why It Matters

The findings reveal that skill‑enhanced agents may overstate their real‑world reliability, urging developers to improve retrieval and refinement before deploying them in production environments.

Key Takeaways

• 34,198 real skills tested across Claude, Kimi, Qwen agents.
• Pass rates fell from 55.4% to 38.4% as conditions realistic.
• Weaker models dropped below no‑skill baseline, e.g., Kimi 19.8% vs 21.8%.
• Hybrid search improved Recall@3 by 18.7 points, yet overall gains modest.

Pulse Analysis

The promise of modular "skills"—structured text files that encode domain‑specific workflows—has driven rapid adoption in agentic AI platforms since Anthropic’s 2025 rollout. Early benchmarks like SKILLSBENCH handed agents pre‑selected, task‑specific skills, effectively giving them a step‑by‑step cheat sheet. This artificial advantage painted an overly optimistic picture of how agents could leverage external knowledge, masking the complexities of real‑world retrieval where skill libraries are noisy, incomplete, and uncurated.

In a comprehensive study led by UCSB, MIT CSAIL, and the MIT‑IBM Watson AI Lab, researchers assembled a corpus of 34,198 permissively‑licensed skills from skillhub.club and skills.sh. They then subjected Claude Opus 4.6, Kimi K2.5, and Qwen 3.5‑397B to six increasingly realistic scenarios, from forced skill loading to autonomous search across the entire collection. Across the board, performance degraded steadily; Claude’s best pass rate of 55.4% dropped to 38.4%, while Kimi and Qwen fell below their no‑skill baselines. The analysis identified three choke points: agents often fail to select relevant skills, retrieval methods achieve only 65.5% Recall@5, and adaptation of generic skills to specific tasks remains elusive.

The implications for enterprises are clear. Relying on skill‑enhanced agents without robust retrieval and refinement pipelines can lead to brittle automation that underperforms in production. The study’s modest gains from a hybrid search strategy and task‑specific refinement suggest that incremental improvements are possible, but only when the underlying skill set is high‑quality and well‑indexed. Companies looking to embed agentic AI should prioritize building searchable skill ecosystems, invest in adaptive refinement mechanisms, and temper expectations until retrieval reliability matches the promise shown in curated benchmarks.