Embracing Uncertainty with AI Agents: Vulnerability Assessment Using Pydantic AI

Why It Matters

Accurately prioritizing vulnerabilities while flagging uncertain cases reduces false confidence and accelerates patch cycles, a critical need for modern security teams.

Key Takeaways

• Union‑type output lets agents admit uncertainty.
• Reduces hallucinated vulnerability data in automated triage.
• Enables OTEL‑backed audit trail for security reviews.
• Prioritizes critical CVEs with contextual exploitability.
• Improves mean‑time‑to‑patch by focusing resources.

Pulse Analysis

Vulnerability overload is a growing pain point for enterprises that rely on thousands of open‑source dependencies. Traditional scanners flood security teams with hundreds of CVE alerts, many of which lack the contextual detail needed to rank them effectively. AI agents promise to sift through this noise, but when forced into a single rigid schema they often fabricate data to satisfy required fields, creating false confidence and alert fatigue. Introducing union‑type structured output lets the model choose between a full vulnerability report and an explicit "unable to assess" response, preserving integrity and making the triage process auditable.

Pydantic AI provides the tooling to enforce these union schemas at runtime. Developers define a CriticalVulnerability model with strict field constraints—CVSS score thresholds, remediation priority ranges, and detailed exploitability explanations—while a complementary UnableToAssess model captures justification, flagged CVEs, and uncertainty categories. The agent’s output_type can be declared as "list[CriticalVulnerability] | UnableToAssess," enabling the LLM to return the appropriate shape without hallucination. Integrated OpenTelemetry (OTEL) hooks, via Logfire or similar platforms, automatically log each outcome, differentiating confident assessments from uncertain cases and feeding data into dashboards that track mean‑time‑to‑patch (MTTP) improvements.

For CISOs and security operations, this pattern translates into measurable risk reduction. By surfacing only vetted, context‑rich vulnerabilities and clearly marking those that need human review, teams can allocate remediation resources where they matter most, shortening exposure windows. The observable audit trail satisfies compliance requirements and supports continuous model refinement, as engineers can analyze uncertainty patterns and retrain models accordingly. As AI‑driven triage matures, union‑type outputs become a best practice for balancing automation speed with the safety of human oversight.