Hugging Face

Apriel-H1: The Surprising Key to Distilling Efficient Reasoning Models

ServiceNow‑AI converted its 15 B attention‑based reasoning model into a hybrid Mamba architecture, achieving 2.1× throughput with negligible quality loss. The breakthrough came from distilling on the teacher’s high‑quality SFT reasoning traces rather than generic pretraining data, and using reverse KL divergence as the loss. A three‑stage replacement process—identifying low‑impact layers, progressive Mamba substitution, and final SFT fine‑tuning—produced the Apriel‑H1‑15b‑Thinker‑SFT checkpoint, which outperforms the teacher on several reasoning benchmarks while cutting latency.