How AI Covered a Human’s Paternity Leave // Quinten Rosseel

Why It Matters

The deployment proves AI agents can become essential, production‑grade tools that sustain operations during talent shortages, reshaping data‑team workflows across the industry.

Key Takeaways

• AI agent answered 60% of data queries during leave
• Slack replaced web UI for higher adoption
• Real-world eval outperformed synthetic BIRD benchmarks
• Context engineering and metadata reduced latency
• Trust building essential for skeptical business users

Pulse Analysis

The unexpected test of an AI analyst during a head of data's paternity leave highlights how quickly agentic systems can become mission‑critical. In a logistics SaaS firm with a lean 2.5‑person data team, the chatbot dubbed “Wobby” fielded roughly 60 % of internal data questions, proving that a well‑engineered agent can fill staffing gaps without sacrificing response quality. This real‑world deployment underscores a broader industry shift: companies are moving from experimental prototypes to production‑grade agents that directly support business decision‑making. The success also sparked interest from other product teams seeking similar automation.

The project also revealed why conventional benchmarks such as the BIRD score can be misleading. Wobby’s team built a custom evaluation pipeline that injected live business queries, exposing failure modes that synthetic tests missed. Technical refinements—context‑aware prompting, rich metadata tagging, and latency‑focused infrastructure—cut average response time by half and improved answer relevance. These engineering choices illustrate that successful agent deployment hinges on tailoring LLM workflows to the specific data landscape rather than relying on generic performance metrics. The evaluation framework now serves as a template for future agent rollouts.

Beyond the code, the human factor proved decisive. Switching the interface from a web dashboard to Slack aligned the agent with existing collaboration habits, driving rapid user adoption. Structured onboarding and transparent confidence scores helped skeptical analysts trust the system, turning Wobby into a daily partner rather than a novelty. As more enterprises confront talent shortages, the lesson is clear: combining robust technical foundations with thoughtful channel design and change‑management practices is essential for scaling AI agents from pilot projects to reliable business assets. Future iterations will explore multimodal inputs to broaden Wobby’s analytical reach.

Original Description

March 3rd, Computer History Museum CODING AGENTS CONFERENCE, come join us while there are still tickets left.

https://luma.com/codingagents

Thanks to @ProsusGroup for collaborating on the Agents in Production Virtual Conference 2025.

Abstract //

When your head of data goes on paternity leave, you learn whether your AI agent actually works. For a logistics SaaS company with a 2.5-person data team, our AI analyst ""Wobby"" became the unexpected backup, handling 60% of incoming data questions from the business. This talk shares the hard-won lessons from taking an AI agent from concept to daily use. You'll learn why we abandoned our web UI for Slack, why BIRD benchmark scores meant nothing for our actual success, and how we built an eval system that caught real failure modes instead of synthetic ones. We'll cover the technical decisions that mattered: context engineering, metadata design, and latency optimization. We'll also cover the non-technical ones that mattered more: channel design, user onboarding, and building trust with skeptical business users. This is a practitioner's guide to agent deployment. What worked, what failed spectacularly, and what we'd do differently next time.

Bio //

Software engineer focused on AI and ML Engineering. Currently building text-to-SQL agents at Wobby and advising early-stage startups on their AI/ML products. Hosting meetups through the Belgian AI & Data Science community in Brussels.

Expertise:

- Python Software Engineering

- Agentic Systems & LLM Workflows

- ML(ops) & Scalable Data Pipelines

Professional Interests:

- Data modelling & Search Engines

- ML Engineering, MLOPS & ML Platforms

- LLMs & Agents

- Domain Expert Collaboration

A Prosus | MLOps Community Production