RL without TD Learning

Summary

The episode introduces a novel off‑policy reinforcement learning algorithm that replaces temporal‑difference learning with a divide‑and‑conquer paradigm, dramatically reducing error accumulation by using logarithmic Bellman recursions. Seohong Park explains how the method leverages the triangle‑inequality property in goal‑conditioned RL, employing a subgoal proposal network to efficiently select intermediate states from the dataset, making the approach scalable to continuous, high‑dimensional tasks. Experiments on long‑horizon benchmarks such as Maze2D and Ant‑Maze show substantially higher success rates, faster convergence, and robustness compared to TD‑based baselines, highlighting divide‑and‑conquer as a promising third paradigm for value learning.