Why AI Still Struggles to Defend Against Cyberattacks Even in the Age of Mythos

Why It Matters

The gap shows that current AI cannot yet replace human analysts for threat hunting, exposing enterprises to AI‑enhanced attacks and prompting urgent investment in more robust SecOps solutions.

Key Takeaways

• 11 top AI models failed to reliably detect threats in benchmark
• Best model flagged only ~45% of malicious events, missing majority
• AI struggled with large log volumes and subtle attack signatures
• Indian security firms poised to lead SecOps amid AI challenges

Pulse Analysis

The Simbian.ai benchmark shines a light on a paradox in the AI‑security arena: while generative models like Anthropic's Mythos can craft convincing phishing emails or automate exploit code, they stumble when asked to sift through massive, unstructured log data. By feeding raw logs containing up to 135,000 entries and letting the models "figure it out," the test mimics the real‑time pressure faced by SOC analysts. Even Claude Opus 4.6, the top performer, identified less than half of the attack stages, underscoring that current LLMs lack the contextual reasoning and prioritization needed for effective threat hunting.

These results have immediate business implications. Enterprises that have begun to rely on AI‑driven security tools may be overestimating their protective capabilities, leaving a blind spot for sophisticated, AI‑generated attacks. The study also warns that open‑source models, which trail commercial releases by only a few months, could soon match or exceed the offensive capabilities of today’s frontier models, democratizing advanced hacking techniques. Companies must therefore balance rapid AI adoption with rigorous validation, integrating human expertise and layered defenses rather than treating AI as a silver bullet.

India's position as a global SecOps hub adds another layer of relevance. Major Indian firms such as TCS, Infosys, Wipro and HCL manage security for thousands of clients worldwide and are quick to adopt emerging technologies. Their leadership could accelerate the development of AI‑augmented defense solutions tailored to the unique challenges highlighted by the benchmark. However, until AI can reliably parse massive log volumes and detect subtle threat signatures, the industry will continue to rely on skilled analysts to bridge the gap, making talent acquisition and upskilling a top priority for security leaders.