PREreview of Hexagon-MLIR: An AI Compilation Stack For Qualcomm's Neural Processing Units (NPUs)
- Published
- DOI
- 10.5281/zenodo.24409201
- License
- CC0 1.0
## Summary
I read this paper as a compiler engineer who has spent years on JIT code generation and SIMD vectorization, and who now works on moving ML models from laptop to on-device execution. Hexagon-MLIR is an open-source MLIR-based compilation stack that lowers Triton kernels and PyTorch models onto Qualcomm's Hexagon NPU. The core idea I took away is a generative one: instead of calling into hand-written operator libraries, the compiler treats fusion as a first-class pass, builds "mega-kernels" from long operator chains, and keeps the working set in the NPU's Tightly Coupled Memory (TCM) to avoid round-tripping data through DDR. Triton kernels go through triton-to-linalg into Linalg, PyTorch models through Torch-MLIR, and the pipeline then applies fusion, tiling for the DDR/TCM hierarchy, multi-threading across HVX contexts via MLIR's Async dialect, double buffering to overlap DMA with compute, and vectorized math libraries.
I found the paper unusually readable for a compiler paper. I could follow a single softmax and a single GELU kernel through every stage — fusion, tiling, multi-threading, double buffering — with the IR shown at each step. That worked example is the paper's best feature, and it is what let me evaluate the design rather than just the claims.
## What I found strong
First, I think the fusion-first framing is the right call, and the authors argue it well. Operator libraries are peak-performance for single ops but act as opaque barriers: data has to round-trip to DRAM between calls, and every new operator needs careful manual implementation. For the long tail of fusion patterns in modern transformers, that does not scale. Generating specialized kernels for arbitrary operator chains is the only approach that keeps up with how fast models change, and I agree with that judgment from my own experience with JIT codegen, where the same tension between hand-tuned kernels and generated code shows up constantly.
Second, I liked the two-stage design of the double-buffering pass. Stage 1 does the structural software pipelining — synthesizing the ping-pong flow, the prologue prefetch, and the two mutually exclusive sub-kernels — while Stage 2 materializes the actual asynchronous DMA transfers and synchronization. Separating legality and scheduling from transport semantics is clean compiler design, and it keeps the transformation deterministic and hazard-free, which matters a lot when the pass is rewriting memory operations.
Third, the multi-threading design is principled. Forming virtual threads with scf.forall and then lowering them to the Async dialect's fork-join pattern keeps parallel semantics declarative and analyzable until late in the pipeline, instead of baking in threads early. Lowering to LLVM coroutines with the async runtime is a sound choice. I also appreciated that the paper reports the overhead honestly: below 16K elements, multi-threading is slower than single-threaded execution because of token management and barrier costs, and the gains only appear past 32K elements.
## My main concerns
1. I could not tell what the speedups are measured against. Table 1 reports large numbers — 63.9x for GELU in float16, 46.5x for RMS-norm, 40.6x for a 2D vector add — but I could not find a clearly stated baseline. If the baseline is a scalar, unvectorized implementation, then most of that gain is HVX vectorization doing what vectorization does, and the number says little about the compiler's own contribution. I would like to see each kernel's speedup broken down against a defined baseline, and ideally against a straightforward hand-written HVX implementation, so I can tell how much of the gain comes from fusion, tiling, and double buffering specifically.
2. There is no comparison against Qualcomm's own commercial toolchain or libraries. The paper says Hexagon-MLIR "complements" the commercial toolchains, but the question every practitioner will ask is: how close does the generated code get to the hand-optimized library, and what do I give up by choosing the open stack? I think the paper would be substantially stronger with even one kernel compared against the vendor library or QNN path. Without that, I cannot judge whether the deployment-speed advantage the authors claim is worth any performance gap.
3. The evaluation is on isolated kernels, not end-to-end models. The motivation section talks about LLMs and Mixture-of-Experts architectures, but the results cover GELU, RMS-norm, SiLU, flash attention, vector add, and an exponent series. I would like to see at least one end-to-end model, or one realistic subgraph, compiled through the full pipeline with latency and memory numbers. A mega-kernel that wins on GELU may behave very differently when fusion decisions interact across a whole transformer block.
4. I found no analysis of numerical accuracy. The pipeline uses fast-math flags and polynomial approximations for exp, and the math-library section explicitly trades controlled numerical error for throughput. That trade is standard practice, but I would expect at least a short error analysis — for example, the maximum relative error of the approximated exp against the exact function, and whether any end-to-end model output was checked. For a stack aimed at real deployment, silent accuracy loss is the failure mode I worry about most.
5. The tile sizes look hand-chosen, and there is no autotuning story. The GELU example tiles at 262,144 elements with 65,536-element thread partitions, and the multi-threading speedup dips at 1M elements, which the authors attribute to TCM and DMA queue pressure and suggest "re-tuning could be beneficial." I agree — but I would like to know how a user is supposed to do that tuning today. Is there a cost model, a search, or is it manual? For a compiler whose value proposition is automated deployment, the tuning story matters.
## Smaller points and questions
- I noticed the paper states that convolutions and matrix multiplications are identified early and sent to a dedicated matmul engine, with linalg.pack/unpack handling layout. That is reasonable, but it means the most compute-heavy ops bypass the pipeline the paper describes. I would like a sentence or two on what fraction of a typical model's runtime that engine covers, so I can judge how much of the workload the presented passes actually touch.
- The semantics-preservation section is careful and honest about quantization weakening strict preservation, and I appreciated that. I would extend the same care to the fast-math approximations discussed above.
- The related-work section covers TVM, Glow, IREE, and Neptune, but I would have liked a direct comparison with IREE, which is the closest analogue — another MLIR stack with a hardware abstraction layer. A short discussion of where Hexagon-MLIR's design diverges from IREE's HAL approach would help me place this work.
- A few presentation nits: one listing reference renders without a number in the text, and the figure numbering in the results section does not match the order the figures are discussed. Neither affects the technical content.
## Overall assessment
I think this is a solid, honest systems paper, and I would recommend it to anyone building compilers for NPUs or on-device ML. Its contribution is not a single new algorithm but a coherent, open pipeline with a clear design philosophy — generate fused kernels, keep data in fast local memory, overlap movement with compute — demonstrated with unusually transparent worked examples. My main reservation is the evaluation: without a stated baseline, a comparison to the vendor library, an end-to-end model, and an accuracy analysis, I can verify the mechanism but not the magnitude of the win. I hope the authors address those points as the work progresses, because the design deserves a stronger evidentiary case.
Competing interests
The author declares that they have no competing interests.
Use of Artificial Intelligence (AI)
The author declares that they used generative AI to come up with new ideas for their review.