#155 - Connor Leahy - "We Don't Know How It Works": An AI Engineer's Warning

The episode opens with a stark reminder: despite rapid progress, we still lack a scientific theory of intelligence. Both the human brain and modern neural networks are described as "black boxes"—vast collections of numbers whose inner workings elude clear explanation. This uncertainty fuels a sense of magic around large language models, as engineers can produce astonishing outputs without truly understanding the underlying processes. For business leaders, the takeaway is clear: deploying AI without grasping its limits can expose organizations to unexpected behavior and hidden failure modes.

Connor traces the turning point to the 2017 transformer architecture, which turned deep learning from a niche tool into a universal pattern learner. By scaling data and compute, transformers evolved from simple word‑level predictors to the foundation of chatbots, image generators, and voice models. He recounts his early open‑source efforts with Llama AI, aiming to democratize research and improve safety, only to discover that maintaining transparent, community‑driven models became untenable as commercial pressures and safety concerns grew. The discussion highlights how the transformer’s attention mechanism, while mathematically simple, creates billions of hidden parameters that no one fully interprets, underscoring the gap between capability and comprehension.

The conversation culminates in a warning about loss of control. Connor argues that humanity may relinquish strategic oversight long before any catastrophic AI event, as systems become too complex to steer directly. This prospect has concrete implications for enterprises: robust governance, continuous monitoring, and investment in explainable‑AI research become essential. Companies leveraging AI clouds like Iron must balance performance gains with ethical safeguards, ensuring that rapid innovation does not outpace the ability to manage emergent risks. The episode serves as a call to action for leaders to prioritize transparency and accountability in AI strategy.