Papers.Data.Code
Paper

NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction

NCP Team, Jiaqi Cao, Chiyu Chen, Shuang Cheng, Xu Cheng, Beiya Dai, Yufan Feng, Kewen Ge

Joint token+concept pretraining replaces token-only supervision with discrete latent concept prediction that is fed back into the decoder.

arXiv ↗Sep 9, 2026Code ↗

  • llm
  • autoregressive language modeling
  • representation learning
  • vector quantization
Detailed pipeline diagram showing encoder, concept module, quantization, and decoder architecture
Detailed pipeline diagram showing encoder, concept module, quantization, and decoder architecture

What it is

A latent-space LM that jointly predicts next tokens and multi-token discrete concepts, then feeds predicted concepts back into decoding for faster, better pretraining. Instead of only next-token prediction, it adds next-concept prediction over a learned product-quantized latent vocabulary; predicted concepts guide decoding while preserving standard AR generation.

Key results

  • Matches OLMo-3-7B final loss using 51.3% of training tokens
  • Beats OLMo-3-7B by 2.45 pts macro average, +5.99 on GSM8K
  • Improves DFlash2 mean accepted length by 4.17% with negligible overhead

Explicit multi-token latent targets can make large-scale autoregressive pretraining both more compute-efficient and more capable.

One useful ML email a week.

More from Sep 14 – Sep 20

The whole week →

Picked Sep 18

Picked Sep 17

Worth your time, not all of your time.

We track new ML work across papers, code, and data, then keep only the pieces worth a closer look. Selected, not collected.

Papers.Data.Code

The Weekly ML Signal

Selected, not collected.

One useful ML email a week: the papers, repos and datasets worth your attention.

Worth your time, not all of your time.