2 NPU Engines + (HRX + Vulkan) = One Engine, any model — Pure C++, zero Python

[bong-water-water-bong]

2 NPU Engines + (HRX + Vulkan) = One Engine, any model — Pure C++, zero Python

A short design write-up for the MLIR/AIE discussion. The full, dated,
receipt-annotated story lives in the repo journey: https://github.com/1bit-MONSTER/1bit-MONSTER/blob/main/docs/journey.md

The one-line thesis

1bit-MONSTER is a single C++26 binary that runs any model on any of the
accelerators in a consumer AMD SoC — and treats them as lanes of one engine,
not as separate backends you have to pick between. The headline configuration:

one `1bit` binary ── router ──┬─ NPU lane A ── (XDNA 2, context 1) ─┐
                              ├─ NPU lane B ── (XDNA 2, context 2) ─┼─ "2 NPU engines"
                              ├─ HRX lane    ── (iGPU, in-process) ─┤
                              └─ Vulkan lane ── (iGPU, RADV) ───────┘
                                          │
                                          ▼
                              auto-routed, failover, byte-correct decode

No config files, no per-model glue, no Python interpreter anywhere at runtime.
Point it at a GGUF / 1BP / ONNX / Q4NX / safetensors file and it detects the
architecture and routes to whatever hardware is present.

---

The three moving parts

1. Two NPU engines on one XDNA 2 die

The Ryzen AI MAX+ 395 (Strix Halo) has a 50 TOPS XDNA 2 NPU that AMD shipped
locked behind a closed-source runtime (FastFlowLM: 22 proprietary .so files,
209 xclbin bitstreams). We reverse-engineered the whole stack in 4 days and
replaced it with open C++. The NPU decode is now byte-identical to AMD's own
runtime — 1000/1000 tokens, 0 ULP, 0 argmax flips.

The "2 NPU engines" part is a scheduling insight, not new silicon: the amdxdna
driver lets a process open two XRT hardware contexts on the one NPU, each
owning a disjoint slice of the AIE column array. That turns one launch-latency-bound
NPU into two overlapping streams:

configuration	throughput	note
1 stream, full 8-column array	10.3 tok/s	numerically clean (corr 0.9985)
2 streams, 2× half-array contexts	9.8 + 9.9 ≈ 19.6 tok/s	~1.9× aggregate, overlapping launch latencies

Accuracy tradeoff (measured): the half-array contexts run at corr ~0.892
vs 0.9985 for the full-array stream — the fused decode kernels are
single-token effective, and splitting the array cuts numerical fidelity.
~1.9× is the throughput ceiling the split unlocks; the full-array path stays
the numerically-clean default until batch-M lands on both halves.

Because fused decode is ~70% launch-latency-bound, two half-array contexts
overlap their launch latencies and nearly double throughput on the same die.
This is the lever that "2 NPU engines" unlocks — and it's what the router
exposes as two independent NPU lanes.

2. HRX — the honest acceleration lane

AMD's experimental HRX runtime (an IREE-based, lighter subset of ROCm) was
moved from a spawned subprocess into our address space: we dlopen the
bundle's libllama.so/libggml-hrx.so with RTLD_DEEPBIND and drive token-level
generate() in-process. Result: warm decode ~80–87 tok/s in-process vs
~38 tok/s as a subprocess vs ~70 tok/s HIP.

The honest ceiling, measured across six model families: HRX's fused node set
excludes GET_ROWS for most quants, so non-fused models fail closed — and the
engine's route-order failover carries them to Vulkan and still completes the
answer. HRX is an acceleration lane, not the engine. It gets re-engaged the
moment a stable upstream release (or llama.cpp PR #27218) lands.

3. Vulkan — the always-works iGPU lane

The Vulkan backend (ggml_vulkan on RADV, Radeon 8060S / gfx1151) runs the
same models end-to-end at ~26 tok/s (vs ~6 on CPU), matching the CPU
reference token-for-token. Crucially, it also participates in the zero-DMA
shared-buffer substrate: a Vulkan shader reads NPU KV-cache pages via dma-buf
import (relative error 2e-4), so the NPU and iGPU lanes share memory without
host copies.

---

Why it's "one engine" and not "five backends"

• One binary — build/1bit is busybox-style; every server/CLI is a
  subcommand of a single ELF.
• Auto-routing + failover — a cost-ranked router dispatches per model (and
  per token), and every lane has a fallback. If HRX can't fuse a quant, it
  routes to Vulkan; if Vulkan init fails, it fails over to HIP/CPU. The answer
  completes regardless.
• Any model — 566 architecture tokens map 1,946 HuggingFace arch strings;
  100% of text-generation checkpoints on the hub land on an engine token.
  16+ model families (Zyphra, Qwen, Llama, Mistral, Gemma, Phi, DeepSeek,
  BitNet/Bonsai, Whisper, …) with zero per-model code.
• Pure C++ — C++26 at runtime, no interpreter; even the model converter
  (gguf_to_onebp) is a real C++ binary, zero Python in the convert path.

---

Milestones (each is a dated entry in the journey)

• 2026-04-16 → 06-28 — first 1-bit HIP kernels on gfx1151, then the 4-day
  XDNA 2 reverse-engineering sprint that killed FastFlowLM.
• 2026-07 — Q4NX format fully decoded; fused-layer engine goes production
  at 291 tok/s (3× the closed runtime); FastFlowLM fully replaced as default.
• 2026-08 — 35B MoE on the NPU; the one-binary 1bit through-line;
  HRX in-process lane lands (and gets honestly reframed as a lane).
• 2026-09 — NPU decode byte-identical to the real runtime (1000/1000, 0 ULP);
  two-NPU-context split (~1.9×); batched MoE + fused decode → Zaya1-8B at
  21.3 tok/s, FLM-class.

The full sequence — every crash, breakthrough, and measured ceiling — is in
docs/journey.md:
https://github.com/1bit-MONSTER/1bit-MONSTER/blob/main/docs/journey.md

---

Repo: https://github.com/1bit-MONSTER/1bit-MONSTER · GPL-3.0 · pure C++26 · zero Python at runtime.