Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents

Key Takeaways

• Claw-Eval audits every agent action via traces, logs, snapshots.
• Traditional final-output tests miss 44% safety and 13% robustness issues.
• Injected errors cut agent consistency up to 24% while performance stays stable.
• Multimodal video tasks remain the weakest point across evaluated models.
• Dialogue success hinges on precise questioning rather than conversation length.

Pulse Analysis

As enterprises integrate large language models into autonomous agents, the industry has grappled with evaluation methods that only verify end results. Such trajectory‑opaque benchmarks can conceal hidden failures, especially when agents interact with software services, handle multimodal data, or operate under safety constraints. By shifting focus from final answers to the full execution path, researchers can surface hidden risks that would otherwise emerge only after costly deployment.

Claw‑Eval tackles this gap with a structured three‑phase workflow. First, a sandboxed environment is prepared with mock services and necessary files. During execution, the agent’s actions are captured through three independent evidence channels: detailed execution traces, server‑side audit logs, and post‑run environment snapshots. The judge component then scores agents on completion, safety, and robustness, injecting controlled perturbations such as network failures to probe resilience. This granular audit ensures that agents cannot cheat by fabricating outcomes, providing a transparent and repeatable assessment.

The framework’s evaluation of 14 cutting‑edge models uncovers stark deficiencies. Traditional benchmarks overlook 44% of safety violations and 13% of robustness lapses, while error injection reduces consistency by up to 24% despite stable peak performance. Multimodal video tasks emerge as the most challenging, and success in multi‑turn dialogues depends more on the precision of questions than on conversational length. For AI product teams, these insights highlight the need for rigorous, trajectory‑aware testing before scaling agents in mission‑critical applications, signaling a shift toward more reliable and trustworthy AI deployments.