Papers.Data.Code
Paper

Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models

Yubo Wang, Jiarong Liang, Yuxuan Zhang, Xuye Liu, Cong Wei, Yuyu Zhang, Ping Nie, Wenhu Chen

Function-level FIM mid-training replaces arbitrary span masking with PDG-selected function reconstruction plus rationale, using surrounding code as agent-like context.

arXiv ↗Jul 14, 2026Code ↗

  • llm
  • nlp
  • code generation
  • fill-in-the-middle
  • tool use
Method pipeline diagram plus main comparative benchmark gains across models
Method pipeline diagram plus main comparative benchmark gains across models

What it is

Mid-trains code LLMs with function-aware FIM so they learn to use returned context like tool outputs before agent fine-tuning. Instead of random-span FIM or pure left-to-right code pretraining, masks whole functions chosen by PDG plus complexity/inferability scores, and trains the model to reconstruct them with rationale from surrounding code.

Key results

  • Qwen3-8B + SWE-Lego: 35.0 Verified, +3.2 over post-train only
  • Qwen2.5-14B + R2E-Gym: 29.2 Verified, +3.0 over reproduced baseline
  • 14B restores +11.1 LiveCodeBench and +3.9 tau-bench vs post-train only

Training on function-call structure before agent tuning improves tool-return conditioning and reduces the capability loss from agentic post-training.

One useful ML email a week.

More from Jul 13 – Jul 19

The whole week →

Picked Jul 17

Picked Jul 16

Worth your time, not all of your time.

We track new ML work across papers, code, and data, then keep only the pieces worth a closer look. Selected, not collected.

Papers.Data.Code

The Weekly ML Signal

Selected, not collected.

One useful ML email a week: the papers, repos and datasets worth your attention.

Worth your time, not all of your time.