What it is
From-scratch Rust LLM inference stack for AMD MI355X that talks directly to /dev/kfd and serves OLMo 2 without ROCm, HIP, or HSA at runtime. Run and serve LLMs on AMD GPUs through the Linux kernel ABI alone, with Rust implementations of dispatch, memory, code-object loading, collectives, attention, and an OpenAI-shaped API.
Key results
- Serves OLMo 2 via /v1/chat/completions on MI355X with no ROCm runtime
- Small all-reduce: ~20 µs at 2 KiB vs RCCL ~26 µs on 8 MI355X
- Paged GQA FP4 decode attention hits 4.5× FP16 single-head speed at 1M KV
Shows that a usable AMD LLM serving stack can be built directly on the kernel ABI, without vendor runtimes at execution time.