What it is
Edge-native MoE serving for local hardware that adapts expert placement and CPU/GPU execution to available bandwidth and memory. It makes frontier MoE models practical on laptops and desktops. Instead of fixed offloading or static expert placement, it continuously adapts MoE execution to machine bandwidth and memory: double-buffered prefill, semantic state reuse, shared expert cache, and a q* split of misses between GPU fills and CPU execution.
Key results
- 1.5-2.3x higher decode throughput than edge baselines on RTX 5090
- Worst-case TTFT stays under 44 s; every baseline exceeds 150 s somewhere
- 39.3 tok/s for a 35B model on an 8GB RTX 4060 laptop, above Codex median 33
Edge MoE serving works much better when expert movement and execution are adapted online to hardware bandwidth and agent-style context edits.