AI Benchmarks: What Jellyfish Learned From Analyzing 20 Million PRs

Why It Matters

Engineering leaders can use Jellyfish’s large-scale, joined dataset to set realistic adoption targets, measure real productivity changes, and spot trade-offs as teams scale AI tooling — helping firms avoid over‑reliance on anecdotal claims. Its benchmark provides a practical baseline for planning licensing, training, and governance around AI-assisted development.

Summary

Jellyfish analyzed telemetry from 1,000 customers — covering about 200,000 developers and 20 million pull requests — to benchmark AI adoption and impact in software engineering. The company links IDE and agent usage (e.g., Copilot) to task and source-control data to measure who has access, weekly active usage, and frequent usage patterns; median tool adoption sits around the low-60s percent of engineers. Jellyfish finds that measurable productivity gains track with frequent, not just occasional, tool use, but warns of side effects and gaps between hype-driven expectations and real outcomes. The firm has published an evolving AI Engineering Trends Benchmark and will update it monthly with new cuts of the data.