Papers.Data.Code
Dataset

SWE-bench Science

OpenMOSS-Team

Contains 119 software-engineering tasks from scientific-computing repos, packaged as Harbor/Pier bundles for coding-agent evaluation.

Hugging Face ↗Aug 14, 2026Code ↗

  • llm
  • coding-agents
  • software-engineering
  • scientific-computing
Headline benchmark comparing multiple models' Pass@1 accuracy, core comparative result
Headline benchmark comparing multiple models' Pass@1 accuracy, core comparative result

What it is

Benchmark dataset for evaluating coding agents on scientific software-engineering tasks with isolated runtimes and separate verifiers. 119 text-defined coding tasks across 20 scientific domains, each paired with a pinned runtime Docker image and a separate verifier image; released as CSV/JSONL Harbor/Pier task bundles.

Key results

  • 119 tasks across 20 scientific domains
  • 96 unrestricted tasks + 23 restricted-license tasks
  • 119 runtime images and 119 verifier images, linux/amd64

Useful for benchmarking coding agents on long-horizon bug-fixing and implementation tasks in scientific codebases.

One useful ML email a week.

More from Aug 17 – Aug 23

The whole week →

Picked Aug 21

Picked Aug 20

Picked Aug 18

Worth your time, not all of your time.

We track new ML work across papers, code, and data, then keep only the pieces worth a closer look. Selected, not collected.

Papers.Data.Code

The Weekly ML Signal

Selected, not collected.

One useful ML email a week: the papers, repos and datasets worth your attention.

Worth your time, not all of your time.