Can LLMs Really Prioritize AppSec?

Why It Matters

Prioritization failures can leave critical bugs unaddressed, increasing breach risk and inflating remediation costs for organizations.

Key Takeaways

• LLM-based app security tools struggle with prioritizing vulnerabilities.
• Developers rarely fix all scanner-reported issues, need triage.
• LLM outputs are non-deterministic, raising consistency concerns for security.
• Traditional SAST scanners provide deterministic, rule‑based findings that developers trust.
• Overlooked issues could leave critical security gaps unaddressed.

Summary

The video questions whether large language models (LLMs) can effectively prioritize application security findings, contrasting them with established static analysis scanners.

The speaker notes that LLM tools often generate high‑quality code suggestions but fall short on triaging vulnerabilities. Developers typically ignore the majority of scanner alerts, so a tool must rank issues by risk. Moreover, LLM outputs are inherently non‑deterministic, leading to inconsistent recommendations, whereas traditional SAST solutions deliver deterministic, rule‑based results.

“Developers won’t fix a hundred results,” the presenter asserts, emphasizing the need for actionable prioritization. He also warns that “LLMs are non‑deterministic, which makes me concerned about consistency,” highlighting potential blind spots that could be missed.

For security teams, the takeaway is clear: relying solely on LLMs may expose gaps, and a hybrid model that combines deterministic scanners with LLM‑driven remediation guidance is advisable. The discussion underscores the business risk of unprioritized vulnerabilities and the importance of maintaining reliable, repeatable assessment processes.

Original Description

Application security isn’t just about generating findings. It’s about prioritizing the right ones.

In this clip, Kalyani Pawar highlights a core challenge with LLM-driven security tools: prioritization. Developers rarely fix every issue surfaced by a scanner. When a tool produces 100 findings, AppSec teams must help determine which ones truly matter. Without strong prioritization, volume becomes noise.

She also raises a deeper concern: determinism. Traditional SAST tools are largely rule-based and consistent in their outputs. LLMs, by design, are non-deterministic. That variability may introduce inconsistency in results — and potentially overlooked vulnerabilities.

The implication isn’t that LLMs are unusable. It’s that reliability and prioritization are foundational to trust in security tooling.

If your AI-powered scanner produces different outputs under similar conditions, how should teams validate and operationalize those findings?

Subscribe to our podcasts: https://securityweekly.com/subscribe

#AppSec #AIinSecurity #DevSecOps #SecurityWeekly #Cybersecurity #InformationSecurity #AI #InfoSec