OpenAI Board Member Zico Kolter on the Real Risks of Frontier AI

Why It Matters

Effective safety oversight ensures that rapidly advancing AI models are deployed responsibly, protecting businesses and the public from emerging misuse and reliability threats.

Key Takeaways

• OpenAI’s Safety & Security Committee reviews every major model release.
• Bigger models don’t automatically become more robust or safe.
• Preparedness frameworks set thresholds for catastrophic risk mitigation.
• Safety must evolve alongside expanding AI control surfaces.
• Explicit safety layers, not size, drive model robustness.

Summary

In this interview, Zico Kolter, chair of OpenAI’s Safety and Security Committee, explains how the board oversees model development and release. The committee functions like an audit board, meeting with internal safety teams, reviewing third‑party reports, and can delay a launch if standards aren’t met. Kolter stresses that safety governance is now as essential as financial oversight for AI firms. Kolter outlines OpenAI’s internal safety architecture—systems, preparedness, alignment, and policy teams—and describes the public “preparedness framework” that enumerates thresholds for bio, cyber, and self‑improvement risks. While model capabilities have improved, he notes that robustness and resistance to manipulation have not kept pace without dedicated safety engineering. He cites concrete examples: a red‑team competition that generated 1.8 million attack attempts, and the surprising simplicity of modern AI—often just a few hundred lines of Python, with complexity emerging from training data. Kolter warns, “you can’t just trust models to get safer by getting bigger,” emphasizing the need for layered monitoring and explicit safety training. The discussion signals that AI companies must institutionalize safety committees, continuously invest in safety stacks, and align governance speed with rapid capability growth. Failure to do so could expose enterprises and societies to escalating misuse and systemic risks.

Original Description

What actually happens before a frontier AI model gets released — and who decides whether it is safe enough? In this episode of The MAD Podcast, Matt Turck sits down with Zico Kolter — OpenAI board member, Head of the Machine Learning Department at Carnegie Mellon, and co-founder of Gray Swan — for a deep conversation on the real risks of frontier AI. They discuss how OpenAI’s safety oversight works before major model releases, why more powerful models do not automatically become safer, how jailbreaks and prompt injection expose real weaknesses in AI systems, why AI agents dramatically expand the attack surface, and where frontier AI is headed next. A clear, practical discussion on OpenAI, AI safety, AI security, AI agents, frontier models, red teaming, reinforcement learning, and the future of AI governance.

Zico Kolter

Website - https://zicokolter.com

LinkedIn - https://www.linkedin.com/in/zico-kolter-560382a4

X/Twitter - https://x.com/zicokolter

The Machine Learning Department at Carnegie Mellon University

Website - https://www.ml.cmu.edu/

X/Twitter - https://x.com/mldcmu

Matt Turck (Managing Director)

Blog - https://mattturck.com

LinkedIn - https://www.linkedin.com/in/turck/

X/Twitter - https://x.com/mattturck

FirstMark

Website - https://firstmark.com

X/Twitter - https://x.com/FirstMarkCap

Listen on:

Spotify - https://open.spotify.com/show/7yLATDSaFvgJG80ACcRJtq

Apple - https://podcasts.apple.com/us/podcast/the-mad-podcast-with-matt-turck/id1686238724

00:00 Intro

01:32 OpenAI board role and Safety & Security Committee

03:53 How OpenAI reviews major model releases

05:33 OpenAI’s preparedness framework explained

09:46 Are frontier AI models getting safer?

12:33 Why AI safety does not come from scale

15:23 The four categories of AI risk

19:38 Doomerism vs accelerationism in AI

24:11 The six-month AI pause debate

26:20 AI safety as a global effort

28:04 How Zico Kolter got into machine learning

31:05 OpenAI in the early days

34:14 Why Carnegie Mellon became an AI powerhouse

38:43 What Gray Swan does in AI security

40:44 AI safety vs AI security

43:15 The GCG jailbreak paper

49:19 How AI labs responded to jailbreak research

50:19 State-of-the-art AI defenses

52:32 State-of-the-art AI attacks

54:22 Why AI agents expand the attack surface

58:39 Are AI agents ready for production?

59:40 Mechanistic interpretability explained

1:02:31 Will AI be safer in two years?

1:03:46 Reinforcement learning and self-improving models

1:08:09 Do post-transformer architectures matter?

1:09:29 Best research directions in AI now

1:11:00 Zico Kolter’s Intro to Modern AI course

1:14:53 Why modern AI is simpler than people think