What it is
Introduces digital teleoperation: a robot-centric world model that turns hand-pose streams plus one reference image into real-time egocentric robot videos and action-labeled training data. Instead of physical teleop or passive human-to-robot video translation, it generates robot-egocentric videos directly from hand poses, with actions preserved for retargeting. It combines depth-aware skeletal control, human-to-robot pretraining, and distilled streaming inference for real-time use.
Key results
- 40+ FPS real-time generation on a single H100 GPU
- Policies trained only on generated data achieve zero-shot Sim2Real transfer
- Adding digital-teleop data consistently improves real-world success rates
Shows robot training trajectories can be generated interactively by a world model and still transfer to real robots without physical teleoperation.