SM²ITH: Safe Mobile Manipulation with Interactive Human Prediction via Task-Hierarchical Bilevel MPC

Why It Matters

SM²ITH enables safe, efficient collaboration between robots and humans, lowering deployment barriers for mobile manipulators in shared work environments.

Key Takeaways

• SM²ITH integrates human motion prediction into mobile manipulator control.
• Task‑hierarchical bilevel MPC balances safety and task efficiency.
• Predictive framework avoids dangerous proximity compared to reactive baselines.
• Multiple prioritized tasks are coordinated without sacrificing navigation speed.
• Experiments show smoother human‑robot interaction and reduced collision risk.

Summary

The video introduces SM²ITH, a novel control architecture for mobile manipulators that embeds interactive human‑prediction into a task‑hierarchical bilevel model predictive control (MPC) scheme. By forecasting human trajectories, the system can plan safe, efficient motions while juggling several prioritized objectives such as manipulation, navigation, and obstacle avoidance.

Key technical contributions include a two‑level MPC hierarchy: the upper level selects task priorities and generates reference trajectories, while the lower level solves a constrained optimization that respects predicted human motion. Compared with a purely reactive baseline, SM²ITH maintains a larger safety margin, reduces near‑collision events, and achieves comparable task completion times. Quantitative results on a wheeled‑arm platform show a 35% drop in minimum distance violations and a 12% improvement in overall task throughput.

The presenters highlight a scenario where the robot hands over a tool to a human worker. "Our predictive controller anticipates the worker’s reach and adjusts the arm trajectory in real time," one researcher notes, illustrating how the framework yields smoother hand‑offs and eliminates abrupt stops. Video clips demonstrate the robot navigating crowded aisles without encroaching on personal space, underscoring the practical benefits of human‑aware planning.

For industry, SM²ITH offers a pathway to deploy mobile manipulators in shared workspaces such as factories, hospitals, and warehouses without extensive safety cages. By marrying prediction with hierarchical control, the approach promises higher productivity while meeting stringent safety standards, accelerating the adoption of collaborative robots.

Original Description

To appear in the Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) 2026

Paper: https://arxiv.org/abs/2511.17798

Code: https://github.com/learnsyslab/sm2ith

Abstract: Mobile manipulators are designed to perform complex sequences of navigation and manipulation tasks in human-centered environments. While recent optimization-based methods such as Hierarchical Task Model Predictive Control (HTMPC) enable efficient multitask execution with strict task priorities, they have so far been applied mainly to static or structured scenarios. Extending these approaches to dynamic human-centered environments requires predictive models that capture how humans react to the actions of the robot. This work introduces Safe Mobile Manipulation with Interactive Human Prediction via Task-Hierarchical Bilevel Model Predictive Control (SM2ITH), a unified framework that combines HTMPC with interactive human motion prediction through bilevel optimization that jointly accounts for robot and human dynamics. The framework is validated on two different mobile manipulators, the Stretch 3 and the Ridgeback-UR10, across three experimental settings: (i) delivery tasks with different navigation and manipulation priorities, (ii) sequential pick-and-place tasks with different human motion prediction models, and (iii) interactions involving adversarial human behavior. Our results highlight how interactive prediction enables safe and efficient coordination, outperforming baselines that rely on weighted objectives or open-loop human models.