Andrej Karpathy Explains Why AI Agent Skills Fail in Long Workflows

Key Takeaways

• Agent skills cause error propagation in multi-step workflows
• Probabilistic models lead to hallucinations and skipped steps
• Harness engineering adds validation loops and state tracking
• Stripe minions and Anthropic plugins showcase deterministic harnesses
• Reliability gains enable AI in audits and diagnostics

Pulse Analysis

The reliability gap in current AI orchestration stems from the probabilistic foundations of agent skills. Each additional step in a workflow introduces a new failure point, allowing minor inaccuracies to snowball into systemic breakdowns. Industries that demand zero‑tolerance errors—financial auditing, medical diagnostics, regulatory reporting—cannot afford such volatility, prompting a shift toward more deterministic architectures that treat AI components as rigorously as legacy code.

Deterministic harness engineering addresses these shortcomings by wrapping each AI sub‑task in a protective framework. Core mechanisms include state tracking to monitor progress, validation loops that catch and correct deviations, and context isolation that prevents cross‑contamination between parallel agents. Sub‑agent delegation further refines precision by assigning specialized models to narrowly defined tasks. Real‑world deployments, such as Stripe’s “minions” for payment reconciliation and Anthropic’s plugin ecosystem, demonstrate measurable reductions in error rates and latency, proving that structured harnesses can scale without sacrificing accuracy.

For the broader market, harness engineering signals a maturation of AI from experimental prototypes to production‑ready services. Enterprises are beginning to embed these deterministic layers into their automation stacks, achieving compliance‑grade confidence while still leveraging large language models for insight generation. Ongoing research focuses on enhancing validation algorithms, optimizing memory management, and integrating graph‑based orchestration, all aimed at tightening the feedback loop between AI output and business rules. As these innovations converge, the promise of reliable, end‑to‑end AI automation becomes a tangible competitive advantage.