The Compute Crunch

Key Takeaways

• Tokens per second now primary bottleneck, not GPU count
• Agentic AI workloads consume 10‑100× more compute per user
• GPU pricing up 48% in two months, spot markets tightening
• Providers impose hidden caps, tiering, and latency trade‑offs
• Roadmaps delayed as firms shelve features to preserve compute budget

Pulse Analysis

The current AI compute crunch stems from an unprecedented surge in token demand. OpenAI’s API traffic has more than doubled in just five months, pushing token‑per‑second capacity to the forefront of infrastructure planning. Unlike earlier bottlenecks that focused on raw GPU counts, today’s limiting factor is how many tokens a system can process reliably. Agentic AI—software that continuously plans, loops, and calls external tools—exacerbates the problem, consuming ten to a hundred times more compute per user session than traditional chatbots.

Market dynamics are reacting sharply. GPU manufacturers have raised prices dramatically; Blackwell‑class GPUs saw a 48% price increase within two months, and spot‑market availability is tightening across the board. Providers such as Anthropic and CoreWeave are imposing multi‑year contracts and steep price hikes, while also layering invisible throttles—rate limits, usage caps, and tiered latency—onto their services. These measures protect margins but erode the user experience, especially as enterprise customers now expect four‑nine (99.99%) uptime, a standard that many AI platforms struggle to meet.

Strategically, compute scarcity is reshaping product roadmaps and competitive positioning. Companies are forced to prioritize features that fit within finite compute budgets, leading to the postponement or cancellation of high‑profile projects like OpenAI’s Sora. Simultaneously, firms are leveraging compute capacity as a differentiator, framing it as a competitive moat in investor communications. As AI adoption accelerates, the ability to secure and efficiently allocate compute will become a decisive factor in market leadership, influencing everything from pricing models to long‑term viability of ambitious AI services.