HoMMI

Learning Whole-Body Mobile Manipulation from Human Demonstrations

We learn whole-body mobile manipulation capabilities directly from human demonstrations, without ANY teleoperation data

Abstract: We present Whole-Body Mobile Manipulation Interface (HoMMI), a data collection and policy learning framework that learns whole-body mobile manipulation directly from robot-free human demonstrations. We augment UMI interfaces with egocentric sensing to capture the global context required for mobile manipulation, enabling portable, robot-free, and scalable data collection. However, naively incorporating egocentric sensing introduces a larger human-to-robot embodiment gap in both observation and action spaces, making policy transfer difficult. We explicitly bridge this gap with a cross-embodiment hand-eye policy design, including an embodiment agnostic visual representation; a relaxed head action representation; and a whole-body controller that realizes hand-eye trajectories through coordinated whole-body motion under robot-specific physical constraints. Together, these enable long-horizon mobile manipulation tasks requiring bimanual and whole-body coordination, navigation, and active perception.

HoMMI Data Collection Interface

HoMMI learns directly from scalable human demonstrations with a portable, in-the-wild data collection interface. We augment UMI grippers with egocentric sensing to capture global context required for mobile manipulation.

Egocentric Sensing Introduces a Larger Human-to-Robot Embodiment Gap

Visual gap: Different egocentric viewpoints & different arms and torso appearances.
Kinematic gap: The robot head is lower than the human, and the robot neck only has 2-DoF.

Cross-Embodiment Hand-Eye Policy

Embodiment-agnostic visual representation: We use a 3D representation for the egocentric view that enables an embodiment-agnostic gripper coordinate frame and masks out embodiment-specific arms and body observations.

Relaxed head action representation: We relax the head action constraint by representing the robot gaze as a 3D look-at point. This allows effective active perception without overconstraining the robot to mimic human head motions exactly.

Constraint-Aware Whole-body Controller

We design a whole-body controller that realizes hand-eye trajectories through coordinated whole-body motion under robot-specific physical constraints.

Results & Findings

We compare with wrist-only, head-only, RGB-only, and w/o active neck baselines.

Finding 1: Wrist-only sensing under-observes global context and bimanual coordination

Finding 2: Head-mounted camera alone is insufficient

Finding 3: Naively adding head RGB and regressing full 6-DoF head pose hurts performance

Finding 4: Active neck control gathers task-relevant context

Finding 5: Our method learns informative attention.

Our method learns informative attention on task-relevant objects, while baselines attend to irrelevant regions.

All Rollouts

FAQ

Acknowledgments

The authors would like to thank Calder Phillips-Grafflin, Aimee Goncalves, and Andrew Beaulieu from TRI for their help with the RB-Y1 hardware setup. Austin Patel for his help with the iPhone data collection app, Maximilian Du for his help with the fisheye camera calibration, and his feedback on the manuscript. Kosei Tanada, Vitor Guizilini, Paarth Shah, Eric Dusel, Sam Creasey, Hillel Hochsztein, Benjamin Burchil, Mengchao Zhang, Mark Zolotas, and Naveen Kuppuswamy from TRI for their helpful discussions. Phoebe Horgan, Allison Henry, Richard Denitto, Maya Angeles, Samuel, Owen Pfannenstiehl, Mariah Smith-Jones, Matthew Tran, and Gordon Richardson from TRI for their help with data collection. Yifan Hou, Huy Ha, Hojung Choi, all REALab members, and Jinghan Sun for their helpful discussions and feedback on the manuscript. This work was supported in part by the NSF Award #2143601, #2037101, and #2132519, and Toyota Research Institute. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.

Citation

	
@misc{xu2026hommilearningwholebodymobile,
	title={HoMMI: Learning Whole-Body Mobile Manipulation from Human Demonstrations}, 
	author={Xiaomeng Xu and Jisang Park and Han Zhang and Eric Cousineau and Aditya Bhat and Jose Barreiros and Dian Wang and Shuran Song},
	year={2026},
	eprint={2603.03243},
	archivePrefix={arXiv},
	primaryClass={cs.RO},
	url={https://arxiv.org/abs/2603.03243}, 
}