Abhishek Thakur

Architecture and Data Trump Size in Small Language Models

In this AI Chitchat episode, we sit down with Fernando from Liquid AI to unpack the rise of small language models (SLMs) and why architecture + data now matters more than raw parameter counts. We cover hybrid transformer/convolutional designs, neural architecture search (NAS), and what it takes to make models smaller, faster, and efficient enough to run locally on consumer GPUs or modern laptops-without sacrificing capability. Fernando shares hard-won lessons from pre-training and post-training: why data quality is king, when optimizers like Adam vs. Muon shine (architecture-dependent!), and practical methods such as layer freezing (Spectrum) and LoRA trade-offs. We dig into distillation (token-prob vs. hidden-state matching), the reality of RLFT for SLMs, and where on-device, agentic AI is heading next: think private, offline assistants that actually manage your email, calendar, research, and more. https://t.co/nv6t6QldiR