Auto-Diagnosing Kubernetes Alerts with HolmesGPT and CNCF Tools

Why It Matters

The approach proves that well‑crafted runbooks, not larger models, drive reliable, cost‑effective automation of Kubernetes alert handling, delivering measurable productivity gains for SRE teams.

Key Takeaways

• Runbooks cut tool calls from 16 to 2 per alert.
• Investigation cost $0.04, about $12 monthly.
• Deduplication reduces 40 daily alerts to 12 investigations.
• Model choice less impactful than runbook guidance.
• Hybrid self‑hosted/managed setup enables seamless model swaps.

Pulse Analysis

The rise of large language models (LLMs) has sparked interest in automating incident response, but STCLab’s experience shows that the real lever is operational knowledge encoded in runbooks. By integrating HolmesGPT with their existing observability stack—OpenTelemetry, Mimir, Loki, Tempo, and Robusta—the team let the LLM follow a ReAct loop that selects the right tool, reads the output, and decides the next step. The runbooks act as a metadata layer, telling the model which data sources are available in each namespace, preventing wasted queries to missing logs or traces. This disciplined guidance reduced the average number of tool calls per alert from sixteen to just two, dramatically cutting latency and cloud‑compute expense.

Cost efficiency emerged as a secondary benefit. Each automated investigation runs at roughly $0.04, translating to about $12 per month for the entire pipeline. The hybrid deployment—self‑hosted models for staging and managed APIs for production—allows seamless model swaps without touching the surrounding glue code, a 200‑line Python playbook that handles deduplication, Slack threading, and routing. The team’s controlled tests demonstrated that the same model scored 4.6/5 with runbooks versus 3.6/5 without, underscoring that operational context outweighs raw model size.

For SRE organizations looking to scale alert triage, the lesson is clear: invest in granular, namespace‑aware runbooks and a robust integration layer before chasing ever‑larger LLMs. The architecture is portable—future extensions like eBPF metrics from CNCF’s Inspektor Gadget can be added without redesign. By marrying observability data, runbook‑driven constraints, and LLM reasoning, teams can achieve faster, cheaper, and more reliable Kubernetes incident resolution.