Black Hat USA 2025 | Pwning User Phishing Training Through Scientific Lure Crafting

Why It Matters

If organizations rely on flawed click‑rate metrics, they may overlook deeper behavioral gaps, leaving critical assets exposed. Rethinking training and measurement can drive more effective, risk‑based security strategies.

Key Takeaways

• Traditional phishing training shows negligible behavior change
• Study covered 20,000+ employees over eight months
• Lure effectiveness varies unpredictably, defying conventional wisdom
• Click metrics often reflect bait quality, not user awareness
• Gamified lure creation can backfire, increasing risk

Pulse Analysis

The Black Hat USA 2025 briefing challenges the long‑standing belief that brief phishing simulations are sufficient to inoculate employees against social engineering. By embedding researchers directly within corporate environments, the study captured real‑world interactions that laboratory tests miss, highlighting a stark disconnect between reported training success and actual user behavior. This evidence suggests that security teams need to move beyond simple click‑rate dashboards and adopt continuous, context‑aware education that addresses the psychological triggers behind phishing decisions.

One of the most striking findings is the chaotic performance of different lure themes. Traditional high‑urgency subjects—such as account suspension notices—did not consistently outperform more mundane hooks like dress‑code updates. This variability underscores that attackers can exploit any narrative that resonates with a specific audience, making it impossible to rely on a static set of training examples. Organizations should therefore diversify their simulated attacks, regularly refreshing content to mirror emerging threat vectors and cultural nuances within their workforce.

Finally, the presentation warns that gamified lure creation, while intended to boost engagement, can inadvertently reinforce harmful patterns. When employees are rewarded for spotting obvious phishing cues, they may develop a tunnel vision that ignores subtler, more sophisticated attacks. A balanced approach that combines realistic scenario training, post‑exercise debriefs, and measurable behavior change—rather than mere click counts—will better equip enterprises to mitigate the evolving phishing landscape.

Original Description

Phishing training has been sold as a silver bullet for twenty years—just show people a few fake emails, teach them what to look for, and they'll magically stop clicking, right? Wrong. Our 8-month, real-world study across 20,000+ employees blows that narrative wide open. We didn't run a controlled lab test. We embedded ourselves in the wild. And what we found was clear: current phishing training doesn't move the needle. Worse, the lures themselves behave chaotically—some bait (like "urgent dress code updates") consistently outperformed others, and not in ways that align with conventional wisdom.

This talk digs into why phishing training metrics are a dangerous mirage—used as both security theater and a flawed defense strategy. We'll dissect how gamified lure creation inside orgs can backfire, how novelty and context collide, and why click rates may say more about the bait than the user. Finally, we'll open the floor to the hard questions: Can internal phish metrics be hacked for good—or evil? Are we designing for behavior change or just measuring clicks? And what does a post-phishing-training world even look like?

By:

Christian Dameff | Co-director, UC San Diego Center for Healthcare Cybersecurity

Ariana Mirian | Senior Security Researcher, Censys

Presentation Materials Available at:

https://blackhat.com/us-25/briefings/schedule/?#pwning-user-phishing-training-through-scientific-lure-crafting-46793