Choosing the Right Model Is Hard. Maintaining Accuracy Is Harder.

Why It Matters

Automating model selection and continuous refinement eliminates costly drift, letting businesses sustain high‑accuracy AI services while reducing compute expenses.

Key Takeaways

• Selecting optimal LLM model is increasingly complex and volatile.
• Fine‑tuning requires extensive data, often hidden in inference logs.
• Pioneer platform automates model deployment, monitoring, and iterative improvement.
• Continuous experiments enable smaller, cheaper models to maintain accuracy.
• Partitioned usage lets agents dynamically choose best model per task.

Summary

Ash Lewis, founder and CEO of Fast Labs, opened the session by highlighting a growing pain point for AI product teams: picking the right large‑language model (LLM) and keeping its performance steady once it’s in production. He noted that the rapid release cycle of new models, shifting benchmark results, and the hidden cost of fine‑tuning make the decision‑making process both opaque and time‑consuming.

Lewis explained that most organizations have valuable inference‑log data that never reaches LLM providers, creating a blind spot for model evaluation and improvement. To close that loop, Fast Labs is launching Pioneer, a platform that first deploys an open‑source model (e.g., Llama, Qwen, DeepSeek) and then continuously monitors real‑world usage, generates synthetic labels, fine‑tunes, and re‑evaluates the model in the background.

A live demo showed the system automatically testing several models on a “weather‑to‑JSON” task, producing accuracy reports and then feeding live inference data back into the training pipeline. Lewis emphasized that this iterative loop not only boosts accuracy over time but also drives down latency and compute costs by allowing smaller models to replace larger ones when feasible.

For enterprises, Pioneer promises a hands‑off solution to model drift, reducing reliance on costly trial‑and‑error model selection and enabling scalable, cost‑effective AI deployments. By automating the entire lifecycle—from selection through continuous refinement—companies can focus on product value rather than the mechanics of LLM maintenance.

Original Description

Ash Lewis (Fastino) Lightning Talk at the Coding Agents Conference at the Computer History Museum, March 3rd, 2026.

Abstract //

Picking a model isn’t the hard part; keeping it useful is, and Ash Lewis is betting on agents to constantly test, swap, and tune models in production because if you “set and forget” your LLM, it’s probably already drifting and getting worse.

Bio //

Ash Lewis is the cofounder and CEO of Fastino, building lightweight, task-specific AI models as an alternative to large, general-purpose LLMs.