SecTor 2025 | Security and Safety Testing for Agentic AI

Why It Matters

Without stateful, context‑aware testing, agentic AI deployments risk catastrophic, hard‑to‑detect failures that can undermine business operations and erode trust.

Key Takeaways

• AI adoption surges; over 80% of large firms use it
• Agentic systems expand threat surface beyond simple input-output
• Current testing remains stateless, missing stateful attack vectors
• Context‑agnostic benchmarks fail to reflect real‑world risks accurately
• Map‑test‑promote framework needed for continuous AI security assessment

Summary

The SecTor 2025 talk highlighted the urgent need for robust security and safety testing of agentic AI systems. Presented by a ServiceNow AI R&D leader, the speaker framed the discussion around the explosive growth of AI adoption—200 million weekly ChatGPT users, half of professional developers using coding assistants, and over 80% of large enterprises integrating AI into core functions—while warning that the complexity of modern agentic architectures is outpacing traditional evaluation methods.

Key insights emphasized that agentic AI no longer operates as a simple input‑output chatbot; instead, it incorporates memory, tool use, and real‑time data streams, creating a vastly larger attack surface. Current testing practices remain largely stateless, focusing on the "front door" of user prompts and ignoring side‑door vectors such as poisoned memory, malicious tool interactions, and environmental manipulation. Moreover, public benchmarks are context‑agnostic, leading teams to overestimate security and underestimate functional degradation when hardening systems.

The speaker illustrated these points with analogies—comparing front‑door testing to a house’s main entrance while side doors remain unsecured—and outlined a five‑area threat‑modeling framework (outcomes, architecture, users/roles, surface vectors, invariance). He advocated for a "map, test, promote" workflow: map risks via detailed threat modeling, test using contextualized benchmarks and automated red‑team exploration, then promote validated findings into regression suites without over‑fitting to specific attack patterns.

Implications are clear: enterprises must shift from static, benchmark‑driven validation to continuous, stateful security assessments that balance safety with functional utility. Adopting the map‑test‑promote methodology will help organizations anticipate tail‑risk scenarios, integrate security into the development lifecycle, and sustain AI deployment at scale.

Original Description

Agentic AI changes the game. If early generative AI systems represented a step change from classic software, agentic AI brings us into a new era. Today we are seeing the early deployment of autonomous and semi-autonomous systems that plan, act, and adapt in open-ended environments. These agents introduce new forms of error and new vectors of exploitation that blur the line between safety failures and security breaches. While major AI labs perform safety and security testing when releasing new models, this testing is often general-purpose and context-agnostic. It is not typically rooted in threat and risk modeling for specific domains or use cases. As a result, high-level claims about model safety and security rarely reflect the risks these systems may pose when embedded in real products and workflows.

This talk focuses on the approach and tools needed for grounded, scalable testing. This starts with threat and risk modeling tied to how agentic systems are used in practice, followed by expert-guided use of two complementary tools: (1) an automated red teaming pipeline that spins up and adapts adversarial and exploratory tests using AI, and (2) DoomArena, an open-source security and safety testing framework for agentic AI that allows for the translation of granular threat and risk modeling into strong, grounded, automated testing.

This talk is for security professionals and enterprise leaders confronting the challenge of understanding and controlling the security and safety risks of genAI systems. It offers a conceptual foundation and practical toolset for testing rigorously at the blurry, high-stakes boundary of security and safety.

By:

Jason Stanley | Head of AI Research Deployment, ServiceNow

Presentation Materials Available at:

https://blackhat.com/sector/2025/briefings/schedule/?#from-prompts-to-plans-security-and-safety-testing-for-agentic-ai-48562