Comments
Write a commentNo comments have been published yet.
## Summary
I read this paper as a compiler engineer who has spent years on JIT code generation and SIMD vectorization, and who now works on moving ML models from laptop to on-device execution. TileLang is a Python-embedded, tile-level programming model for writing high-performance AI kernels on GPUs. Its central idea is one I find compelling: decouple the kernel's dataflow from its scheduling. The programmer writes the dataflow with composable tile operators — `T.copy`, `T.gemm`, `T.reduce`, `T.atomic` — and the compiler takes responsibility for thread binding, memory layout, tensorization, and pipelining, with scheduling primitives (`T.Parallel`, `T.Pipelined`, `T.annotate_layout`, `T.use_swizzle`) available when the defaults are not enough.
The compilation pipeline parses the Python program into a TileLang AST, lowers it to TVM IR, runs a series of optimization passes, and generates LLVM IR, CUDA C/C++, or HIP. Two mechanisms carry most of the design's weight.
First, there is a composable Layout abstraction — a layout function mapping high-level indices to memory addresses, built on IterVars — with a Fragment extension that maps a block-level register allocation down to per-thread register indices and each thread's position within the register file, and a top-down Layout Inference pass that resolves buffer layouts and thread bindings in priority order, with GEMM's strict Tensor Core constraints taking precedence over looser elementwise operators.
Second, there is an automated software pipeline that infers dependencies between copy and compute and inserts `cp.async` on Ampere and TMA plus `wgmma.mma_async` with automatic warp specialization on Hopper, so the user only sets `num_stages`.
The evaluation covers H100, A100, and MI300X (plus RTX 4090 in the GEMM results) on FlashAttention, linear attention, multi-head latent attention, GEMM, and dequantized GEMM, against FlashAttention-3, Triton, cuBLAS/rocBLAS, PyTorch, Marlin, BitsandBytes, FlashInfer, and AITER. I found the appendix genuinely useful: it lists the benchmark shapes for GEMM, FlashAttention, and linear attention (though not for MLA or dequantized GEMM) and prints full kernel implementations for matmul, dequantized matmul, and FlashMLA (Appendix B.2's dequantized example is a W_FP4_E2M1 kernel, not one of the benchmarked INT4/NF4 kernels).
## What I found strong
First, the dataflow/scheduling split is the right abstraction, and the paper argues it well against the right targets. Triton hides thread behavior, memory layout, and address spaces behind automatically generated strategies, which is easy to start with but blocks experts — the paper's example of quantized-weight GEMM needing inline assembly for vectorized datatype conversions is one I recognize from my own work, where the same tension between hand-tuned kernels and generated code shows up constantly.
TileLang keeps Triton's Pythonic tile-level feel while letting an expert declare buffers at each level of the memory hierarchy (`T.alloc_shared`, `T.alloc_fragment`) and drop to thread-level control when it matters. That is the design point I would have chosen.
Second, the Layout Inference mechanism is the most principled part of the paper. Thread binding is the detail that makes GPU kernel programming painful, and the paper's approach — a LayoutMap, a hierarchical priority system, top-down inference from the strictest operator downward — is a clean way to solve it. I liked the worked example where the GEMM output's fragment layout forces the bias buffer's layout to be replicated across threads. This is the kind of inference that, when it works, removes exactly the boilerplate that makes hand-written kernels error-prone.
Third, the Hopper pipeline automation is a real result, not just a design claim. Automatically deriving warp specialization — splitting producers and consumers by threadIdx, using live variable analysis to place mbarrier synchronization — and beating FlashAttention-3 by 1.36× is strong evidence that the automation works on the hardest current hardware. The dequantized GEMM work is the other place where I think TileLang earns its keep: exposing thread-level PTX conversion instructions and Ladder's layout transformation to get 1.04× average over Marlin on W_INT4 A_FP16 and 1.62× average over BitsandBytes on W_NF4 A_FP16 — optimizations the paper credibly argues Triton struggles to express.
## My main concerns
1. The headline MLA number is misleading, and it is the number most readers will remember. On H100, TileLang is reported as a 1075.9x speedup over Torch, and 129.2x on MI300X. But the paper never defines the Torch baseline for MLA — §5.1 describes PyTorch only as 'not fully optimized' — so the 1075.9× figure has no interpretable denominator. The numbers that actually matter are the ones given less prominence: up to 98% of hand-optimized FlashMLA on H100 and 95% of AITER on MI300X. Those are good results, and I would rather the paper lead with them. As written, the 1075.9x figure reads as marketing, and it undermines trust in the more careful numbers elsewhere.
2. On plain GEMM, TileLang matches the vendor libraries — which is honest, but it limits what the paper proves. Against cuBLAS/rocBLAS the speedups are 1.10x on RTX 4090, 0.97x on A100, 1.00x on H100, and 1.04x on MI300X. I respect that the paper reports the 0.97x instead of hiding it. But it also means that for the most common kernel in AI workloads, TileLang's value is programmability, not performance — and on the paper's stated default, part of that parity comes from `T.gemm` delegating to CUTLASS or composable kernels under the hood. I would like the paper to be explicit about how much of the GEMM result is TileLang's own scheduling versus the vendor library it calls into, since the paper's own future work admits the goal is a self-hosting Tile Library that eliminates the CUTLASS and manually wrapped CUDA/HIP dependency.
3. The biggest dequantization number is an apples-to-oranges comparison. The reported maximum of 7.65x over cuBLAS compares a BitBLAS-TileLang W_INT2 A_INT8 configuration against a W_FP16 A_FP16 cuBLAS baseline — different precisions, different arithmetic, different memory traffic. The like-for-like numbers (1.04× average over Marlin, 1.62× average over BitsandBytes) are the honest ones, and they are far more modest. The 7.65x figure should be clearly labeled as a cross-precision comparison or dropped, because as presented it invites the reader to overestimate the gain.
4. I found no compile-time or tuning-time numbers, despite the paper raising the issue itself. Section 4.3 notes that template expansion accounts for approximately 90% of compilation time for TileLang-generated CUDA code (via the Tile Library path) under NVCC 12.8 — a real pain point for anyone iterating on kernels — yet the evaluation reports no end-to-end compile times for TileLang versus Triton or CUTLASS, and no autotuning or search time either. The tile configurations used for each benchmark shape are not listed in the paper, so I cannot tell how much hand-tuning went into the reported numbers or how long a new user would spend reaching them. For a system whose pitch is faster kernel development, development time is part of the evaluation, and it is missing.
5. The evaluation is kernels only, with no end-to-end model and no correctness story. Everything measured is a single operator — attention variants, GEMM, dequantized GEMM. There is no end-to-end model latency, no training step, and no evaluation of the dynamic-shape support the paper claims as a strength for kernel libraries. I also found no mention of numerical verification: the dequantized kernels use low-precision formats and PTX-based fast conversions, and the paper never states whether outputs were checked against a reference. For a stack aimed at production deployment, I would expect at least a short statement of how correctness was validated.
## Smaller points and questions
- I would like to know what happens when Layout Inference cannot resolve a layout. The priority system is described, but there is no discussion of failure modes: does the compiler error out, fall back to a slow default, or silently produce a legal but poor layout? For a system that asks users to trust automatic inference, the fallback behavior matters.
- The paper claims support for dynamic shapes and dynamic parameters, but the benchmark shapes in Appendix A are all static, and the future work section lists dynamic-shape tuning as still to be explored. I think the claim should be scoped to what is actually demonstrated.
- The FlashAttention comparison would be easier to interpret with the sequence-length sweep made explicit in the text. The paper notes FlashAttention-3's fixed tile sizes hurt it at short sequence lengths and that TileLang is close to it at 8k, but the appendix shapes (512–4096) and the discussion do not line up cleanly enough for me to see where the crossover is.
- Hardware coverage is GPU-only in the evaluation, although the conclusion mentions CPUs and NPUs as future backends. That is fine as future work, but I would not describe the system as a general "AI systems" programming model until a non-GPU backend exists.
- A minor presentation point: the GEMM results mention the RTX 4090, which does not appear in the experimental setup's hardware list (H100, A100, MI300X). I assume it was an additional machine, but the setup section should list it.
## Overall assessment
This is a strong systems paper with a design I would like to see adopted more widely. The dataflow/scheduling separation, the Layout Inference mechanism, and the automated Hopper pipeline — including warp specialization — are real contributions, and the FlashAttention and dequantized GEMM results show the automation working on genuinely hard kernels. My reservations are about the evaluation's framing rather than the system itself: the 1075.9x MLA figure and the 7.65x dequantization figure overstate the gains by comparing against an undefined or mismatched baseline, the GEMM parity is partly the vendor library's work, and compile time, tuning effort, and correctness validation are unreported. I would recommend the paper to anyone building GPU kernels or kernel compilers, and I hope the authors tighten the comparisons and add the missing cost numbers, because the design deserves an evaluation as careful as the system itself.
The author declares that they have no competing interests.
The author declares that they used generative AI to come up with new ideas for their review.
No comments have been published yet.