Iterative Gumbel Planning for Continuous Control
NeurIPS, 2026 Accepted
This work continues the EfficientZero line of research on sample-efficient, search-based reinforcement learning. It studies a structural mismatch in continuous control: search evaluates a finite set of candidate actions and returns a discrete distribution, while a Gaussian policy represents a continuous density.
We show that distilling search outputs into a Gaussian policy induces a non-vanishing projection error, inflating policy variance and producing a persistent return gap. Iterative Gumbel Planning (IGP) instead uses a per-dimension factorized categorical policy aligned with the search support. This eliminates the projection error and leaves a bounded discretization error controlled by the grid resolution.
IGP combines Gumbel Top-K sampling without replacement with multiple rounds of search and refinement at the same state. Experiments on DMControl and HumanoidBench demonstrate improved sample efficiency and lower temporal action variance at evaluation.