What it is
Predicts future 3D interaction traces from an image, language, and keypoint history, enabling embodiment-agnostic world modeling and downstream robot policy training. Predict future 3D trajectories of semantic interaction points from a single image, language instruction, and keypoint history, then use those traces for evaluation or downstream robot policy training.
Key results
- Predicts B-spline future trajectories in normalized 3D space with a done head
- Runs from a single image at inference; depth/keypoints can be synthesized
- Includes RoboCasa policy training and evaluation on predicted 3D traces
Shows that world models can plan around semantic 3D interaction traces instead of pixels or robot-specific actions.