Stop Shipping on Vibes — How to Build Real Evals for Coding Agents

Why It Matters

Without reliable evaluations, companies risk deploying ineffective or unsafe code‑generating agents, leading to costly errors and eroding trust in AI solutions. Robust evals provide measurable ROI and accelerate responsible AI adoption.

Key Takeaways

• Current AI coding agents evaluated without rigorous metrics.
• Lack of standardized eval datasets leads to unreliable deployments.
• Braintrust offers tooling to create reproducible AI evals.
• Proper scoring enables measurable improvements and risk mitigation.
• Industry adoption hinges on transparent, data-driven validation.

Pulse Analysis

The surge of AI‑powered coding assistants has transformed software development, promising to write, debug, and refactor code at unprecedented speed. Yet the rapid rollout of these agents often outpaces the establishment of rigorous performance metrics. Developers and product teams are eager to showcase headline‑grabbing demos, but without standardized benchmarks, it’s impossible to compare models or track progress over time. This gap mirrors early stages of other emerging technologies, where hype eclipses hard data, leaving organizations uncertain about the true value of their AI investments.

Jessica Wang, a developer advocate at Braintrust, coined the phrase “shipping on vibes” to describe the prevailing practice of releasing coding agents based on intuition rather than evidence. In many firms, evaluation relies on anecdotal feedback or isolated test cases, which fail to expose systematic bugs, security vulnerabilities, or scalability issues. Without a curated eval dataset and a transparent scoring framework, teams cannot reliably measure precision, recall, or the economic impact of generated code. This uncertainty hampers risk management and makes it difficult to justify continued investment in AI development pipelines.

To move beyond guesswork, Braintrust has introduced open‑source tooling that streamlines the creation of reproducible evaluation suites, integrates automated scoring, and supports continuous benchmarking across model versions. By publishing shared eval datasets, the community can compare agents on common tasks such as unit‑test generation, bug‑fix accuracy, and runtime efficiency. Companies that adopt these practices gain clearer ROI signals, reduce deployment failures, and build stakeholder confidence in AI‑augmented development. As the market matures, rigorous evals will become a competitive differentiator, shaping standards that govern trustworthy coding agents.

Original Description

Jessica Wang (Braintrust) Keynote at the Coding Agents Conference at the Computer History Museum, March 3rd, 2026.

Abstract //

“Shipping on vibes” is how AI breaks and Jessica Wang is calling it out, arguing that without real evals datasets, scoring, experiments you’re just guessing, because the hard truth is most teams don’t know if their agents actually work, they just hope they do.

Bio //

Jess is a Developer Advocate at Braintrust, where she focuses on developer education and tooling around AI evals. She creates technical content on her own social platforms and hosts a podcast on all things tech. Previously, Jess worked at Microsoft, DoorDash, and Warp. Outside of work, she enjoys playing pickleball, tennis, flag football, and ultimate frisbee.