Anthropic Shipped Three Regressions in a Month and Their Evals Didn’t Catch One of Them

Key Takeaways

• Three separate model changes caused Claude Code regressions in April.
• Default reasoning effort shift reduced intelligence but lowered latency.
• Caching bug cleared context each turn, breaking multi-step reasoning.
• Prompt tweak to curb verbosity introduced inconsistent behavior across models.
• User feedback, not internal evals, surfaced the issues.

Pulse Analysis

Anthropic’s recent post‑mortem offers a rare glimpse into the hidden fragilities of large‑scale AI product pipelines. While the firm boasts one of the most sophisticated evaluation frameworks in the industry, the three regressions—effort‑level adjustment, a caching mishap, and a verbosity‑reduction prompt—slipped through its automated checks. This mismatch highlights a broader challenge: benchmark‑driven metrics often capture average performance but overlook edge‑case scenarios that real users encounter, such as complex multi‑turn reasoning or latency‑sensitive workflows.

Each regression tells a distinct technical story. The shift from high to medium reasoning effort traded a modest drop in benchmark scores for faster response times, yet it disproportionately harmed tasks requiring deep, multi‑step analysis. The caching optimization, intended to free stale state, mistakenly erased contextual memory on every turn, effectively resetting the model’s thought process. Finally, the system‑prompt change aimed to curb verbosity but introduced subtle instruction conflicts across model versions, leading to erratic output quality. Together, these issues manifested as inconsistent, hard‑to‑reproduce complaints that internal usage logs and evals initially dismissed as normal variance.

For machine‑learning engineers, the takeaway is clear: robust evaluation must extend beyond static test sets to incorporate live user signals, diverse workload simulations, and continuous A/B monitoring. Building feedback loops that prioritize real‑world performance can catch regressions before they erode trust. Anthropic’s experience serves as a cautionary tale that even well‑funded AI teams need to balance rigorous internal testing with vigilant, user‑driven observability to sustain product reliability.