Papers.Data.Code
Paper

Learning the 4D World through Point Trajectory Forecasting

Byungwoo Jeon, Sunghwan Hong, Dohyeon Kim, Jonghoon Lee, Jungwoo Park, Inhee Lee, Doosan Baek, Jinil Kim

Joint video+trajectory forecasting replaces video-only supervision with persistent 3D point tracks across views, grounding future appearance in shared scene motion.

arXiv ↗Oct 20, 2026Code ↗

  • cv
  • video prediction
  • 3d tracking
  • world models
Detailed pipeline diagram showing reasoner, generator, point tracker method flow
Detailed pipeline diagram showing reasoner, generator, point tracker method flow

What it is

A multi-view world model that jointly predicts future videos and persistent 3D point trajectories, using point-motion supervision to improve 4D forecasting and robot control. Instead of video-only forecasting or per-frame geometry targets, it jointly predicts future videos and persistent 3D point tracks across views, anchoring appearance changes to shared 3D motion and extending the same backbone to action prediction.

Key results

  • VP-WorldBench robot: FVD 67.35 vs 78.52 for VPM video-only
  • VP-WorldBench general: FVD 81.92 vs 101.37 for VPM video-only
  • CMU Panoptic: FVD 128.64, δ3Davg 42.03 vs 150.63, 34.72 for VPM video-only

Persistent 3D point-track supervision makes multi-view future video prediction more geometrically coherent and visually accurate.

One useful ML email a week.

More from Oct 19 – Oct 25

The whole week →

Picked Oct 23

Picked Oct 22

  • Paper
    SolarRenderer: Few-Step Generative Reference Rendering

    Achieves SOTA on in-distribution and 16-type generalization benchmarks, generalizing to 11 unseen structure types; 4-step rendering is 12.3x faster than 50-step sampling.

    arXivOct 20, 2026
    →
  • Repo
    MemTensor/MemT-1.0

    Unlike retrieval-style agent memory, it persists latent session state directly, cutting 128K-history storage about 66×.

    GitHubOct 19, 2026
    →
  • Dataset
    Enderfga/SolarRenderer-Data

    Benchmarks structure-to-video rendering across four control modalities, including unseen control-type splits.

    Hugging FaceOct 15, 2026
    →

Worth your time, not all of your time.

We track new ML work across papers, code, and data, then keep only the pieces worth a closer look. Selected, not collected.

Papers.Data.Code

The Weekly ML Signal

Selected, not collected.

One useful ML email a week: the papers, repos and datasets worth your attention.

Worth your time, not all of your time.