Data Chaz

Train LLMs 3× Faster Without Accuracy Loss

This is a big. Train LLMs ~3× faster with no accuracy tradeoff. @UnslothAI just rolled out a strong set of kernel + data pipeline upgrades that remove real training bottlenecks: padding overhead, unfused ops, and long-context indexing pain. Result: models like Qwen3-4B now train on ~3GB VRAM, at roughly 3× the speed. What changed under the hood: → Fused QK RoPE Triton kernel: ~2.3× faster rotary embeddings, fully compatible with packing → Updated SwiGLU / GeGLU kernels: Int64-safe indexing, long-context training no longer breaks → Clean sample packing: 2.5×–5× higher throughput across FA3, xFormers, and SDPA, identical loss curves → Padding-free batching by default: ~2.1× faster training, ~50% less VRAM, zero accuracy loss Works across SFT, full fine-tuning, and pretraining. This is performance engineering done right: remove waste instead of piling on complexity. Links to announcement + demo below 🧵↓