What it is
A multimodal world model that learns a shared latent state from video and language, then reads it out to text, images, and robot actions with frozen backbone probes. Instead of separate next-token, next-frame, or next-action training, it learns a unified next-state latent from video and language. It combines self-supervised dense video transitions with language-conditioned event transitions, then probes the frozen latent with lightweight decoders.
Key results
- Uses 125K hours of video and 160M event annotations for pretraining
- Training throughput rises from 0.66 to 2.91 samples/sec/GPU, 4.4x vs StarVLA
- Frozen-latent readouts beat similar-sized specialized baselines on text, image, and action
A shared latent trained on dense video dynamics plus language-described events can support multiple downstream modalities without end-to-end task-specific backbone tuning.