What it is
A systematic study of unified multimodal pretraining that maps cross-modal knowledge flow and derives efficient early-joint training recipes. Instead of proposing another multimodal model, it isolates what makes unified pretraining work: asymmetric knowledge flow, complexity-driven synergy, early joint training, and shared attention/norm with modality-specific FFNs.
Key results
- Strong generative performance with only 5% of the compute budget
- Findings validated at scale on multiple 13.5B MoE models trained on 2T tokens
Unified multimodal models work best when modalities are co-trained early and share core attention pathways while keeping some modality-specific capacity.