Paper

AgentGarten: Code Worlds for Evolving Agents

Separates dynamics from rendering: code worlds run in engines, while a shared distilled video model renders observations from geometry in real time.

arXiv ↗Oct 8, 2026Code ↗

  • cv
  • world models
  • simulation
  • video generation
Composite showing diverse worlds, tasks, and evolving agent strategies the work produces
Composite showing diverse worlds, tasks, and evolving agent strategies the work produces

What it is

Builds interactive code-defined worlds by pairing explicit simulators with a shared neural renderer, then studies agents that improve across rounds via inherited playbooks. Instead of choosing between programmable simulators and realistic video world models, it splits them: engines keep explicit state/rules while one real-time neural renderer turns geometry conditions into visuals. It adds Adversarial Forcing to distill a pretrained video model into a few-step interactive renderer.

Key results

  • Agents learn in 4 rounds vs millions for a conventional RL counterpart
  • Shelter-building emerged after 4 rounds in hide-and-seek
  • Seekers learned ramp use after 10 rounds in hide-and-seek

Shows that explicit simulators plus a shared neural renderer can make interactive worlds both programmable and visually rich enough for rapid agent skill evolution.

One useful ML email a week.

Worth your time, not all of your time.

We track new ML work across papers, code, and data, then keep only the pieces worth a closer look. Selected, not collected.

Papers.Data.Code

The Weekly ML Signal

Selected, not collected.

One useful ML email a week: the papers, repos and datasets worth your attention.

Worth your time, not all of your time.