Technology Videos

RAG Evaluation Metrics Tutorial

The video walks through a systematic evaluation of the GraphRAG system, contrasting three retrieval modes—local, global, and hybrid—using the RAGAX benchmark and custom graph-specific metrics. Local mode relies on vector search plus one-hop graph expansion, global draws on LLM-generated community summaries, and hybrid blends both. Ten curated questions spanning factual, relational, and thematic categories are run across all modes, and four standard RAG metrics—faithfulness, answer relevancy, context precision, and context recall—are measured. Additionally, three structural metrics assess entity coverage, graph utilization, and community coherence. Results show hybrid achieving the highest faithfulness score (0.61) versus local (0.183) and global (0.49), with similar leads on relevancy and recall. For factual queries, local and hybrid share the top entity-coverage rate (0.567), while global lags (0.367). Sample questions illustrate the distinctions, such as comparing Sam Altman and Elon Musk on AGI timelines (relational) and extracting a common compute perspective across interviews (thematic). The presenter stresses that “gut feel is not a metric,” underscoring the need for quantitative proof. These findings guide practitioners on mode selection: use local for single-source factual answers, global for broad thematic synthesis, and hybrid for complex queries requiring both precise grounding and breadth. The graph-specific metrics also highlight where graph expansion adds value, informing future improvements and justifying production deployment.