AI Finds Vulns You Can't With Nicholas Carlini

During the hour‑long discussion, Nicholas Carlini explained how large language models have moved from being research curiosities to practical vulnerability hunters. He referenced Anthropic’s February blog post that claimed 500 zero‑day bugs generated by an AI, a headline that sparked industry debate. Unlike earlier attempts that required elaborate scaffolding, today’s models can be pointed at a code base with a short script and produce exploitable inputs. This shift reduces the manual effort traditionally needed for fuzzing and opens a new frontier where AI assists every stage of security testing.

The team’s workflow centers on Claude, a commercial LLM, which they treat as a virtual fuzzer. By compiling targets with AddressSanitizer (ASAN) and feeding the binary to the model, Claude proposes inputs that trigger crashes, providing a reliable oracle for memory‑corruption bugs. This approach uncovered dozens of real issues in projects like Firefox and even a high‑severity SQL injection in the Ghost CMS, complete with a crafted exploit. After each generation, a secondary critique agent evaluates the report, assigns a CVSS‑like score, and filters out false positives before human review.

These experiments demonstrate that AI can dramatically accelerate vulnerability discovery while lowering the expertise threshold required to find bugs. However, Carlini warns that models can hallucinate, so rigorous validation remains essential. As language models continue to improve, security teams are likely to integrate AI‑driven scanning into their DevSecOps pipelines, using automated prompts and critique loops to prioritize high‑impact findings. Organizations should prepare by establishing verification processes, updating threat models, and monitoring AI‑generated advisories to stay ahead of attackers who will increasingly rely on the same technology.