What it is
An end-to-end streaming Transformer for full-duplex audio, video, and text interaction that jointly perceives, reasons, and generates synchronized responses in real time. Instead of cascaded ASR/LLM/TTS/avatar pipelines, one causal Transformer handles audio, video, and text as both inputs and outputs with block-causal streaming, plus a thinker-performer serving split for low-latency generation.
Key results
- ~200 ms model-side signal-to-signal latency at 25 FPS output
- ~550 ms total interaction latency with 350 ms bidirectional network
- Supports streaming units as short as 160 ms at 25 FPS
Shows that full-duplex audio-visual interaction can be learned and served as one causal multimodal model rather than a latency-heavy cascade.