Stanford Online

Stanford AA228 Decision Making Under Uncertainty | Autumn 2025 | Offline Belief State Planning

The lecture introduced offline belief‑state planning for partially observable Markov decision processes, emphasizing that exact POMDP solvers quickly become intractable and motivating scalable approximations. Students were shown how the number of alpha vectors grows exponentially—e.g., a ten‑step horizon can generate 10^338 conditional plans—making exact value iteration impractical for all but trivial problems. The instructor then presented QMDP, an offline method that first solves the fully observable MDP, obtains Q‑values, and then computes a weighted average using the current belief distribution to select actions. A concrete aircraft collision‑avoidance scenario illustrated QMDP’s real‑world relevance, noting that the ACAS X system employs this technique. The discussion also transitioned to alpha‑vector notation, showing that each action’s Q‑values can be treated as an alpha vector, thereby aligning QMDP with standard POMDP representations. Finally, the talk highlighted a curriculum pathway: design decision‑making systems, optimize them, then validate safety‑critical behavior before deployment, underscoring the practical importance of offline approximations and rigorous validation for industry applications.