Stanford AA228 Decision Making Under Uncertainty | Autumn 2025 | Offline Belief State Planning

Why It Matters

Offline approximations like QMDP make decision‑making under uncertainty tractable for safety‑critical systems, while validation courses ensure those policies operate reliably before real‑world deployment.

Key Takeaways

• Exact POMDP solutions become intractable beyond tiny horizons
• Offline approximate methods trade optimality for scalability in practice
• QMDP leverages MDP Q-values weighted by belief distributions
• Alpha‑vector notation unifies QMDP with traditional POMDP frameworks
• Validation courses ensure designed policies behave safely in deployment

Summary

The lecture introduced offline belief‑state planning for partially observable Markov decision processes, emphasizing that exact POMDP solvers quickly become intractable and motivating scalable approximations.

Students were shown how the number of alpha vectors grows exponentially—e.g., a ten‑step horizon can generate 10^338 conditional plans—making exact value iteration impractical for all but trivial problems. The instructor then presented QMDP, an offline method that first solves the fully observable MDP, obtains Q‑values, and then computes a weighted average using the current belief distribution to select actions.

A concrete aircraft collision‑avoidance scenario illustrated QMDP’s real‑world relevance, noting that the ACAS X system employs this technique. The discussion also transitioned to alpha‑vector notation, showing that each action’s Q‑values can be treated as an alpha vector, thereby aligning QMDP with standard POMDP representations.

Finally, the talk highlighted a curriculum pathway: design decision‑making systems, optimize them, then validate safety‑critical behavior before deployment, underscoring the practical importance of offline approximations and rigorous validation for industry applications.

Original Description

For more information about Stanford’s Robotics and Autonomous Systems graduate programs, visit: https://online.stanford.edu/programs/robotics-and-autonomous-systems-graduate-certificate

November 13, 2025

This lecture covers offline belief-state planning.

To learn more about enrolling in this course, visit: https://online.stanford.edu/courses/aa228-decision-making-under-uncertainty

To follow along with the course schedule and syllabus, visit: https://aa228.stanford.edu/

Sydney Katz is a postdoctoral researcher at Stanford's Intelligent Systems Lab.