What it is
A semantic action harness lets VLMs directly control robots via discrete, interpretable moves, enabling zero-shot frontier-model control and cheap fine-tuning of small VLMs. Instead of regressing embodiment-specific motor actions or calling opaque skills, VLMs emit fine-grained semantic action units that deterministic interpreters ground on each robot.
Key results
- Outperforms representative agentic paradigms across tasks and embodiments
- Beats representative VLA baselines under controlled generalization tests
- Shows strong sim-to-real transfer with small VLMs fine-tuned in a few GPU-hours
A compact semantic action interface can preserve VLM reasoning while still supporting direct, fine-grained physical control.