Saltar al contenido principal

Escribir un comentario

PREreview del Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler

Publicado
DOI
10.5281/zenodo.24409417
Licencia
CC0 1.0

## Summary

I read this paper as a compiler engineer who has spent years on JIT code generation and SIMD vectorization, and who now works on moving ML models from laptop to on-device execution. Triton-distributed extends the Triton compiler with native support for overlapping computation, memory access, and communication in distributed AI workloads. The core idea is that communication stops being a library call: OpenSHMEM primitives are integrated into the compiler, so a programmer writes both the compute kernel and the communication logic in Python, and the compiler lowers the communication part to NVSHMEM or ROCSHMEM while the compute part goes through the ordinary Triton stack to LLVM IR and then PTX or AMDGCN.

The model rests on three concepts: symmetric memory (every rank allocates an identically sized buffer, with remote buffers accessed only through primitives), signal exchange (signals in symmetric memory, manipulated with set, add, check, and spin-lock operations), and async-tasks (communication and computation as parallel tasks synchronized through signals). My anchor example is the inter-node AllGather GEMM: separate threadblocks handle intra-node dispatch and inter-node transfer, the GEMM kernel adds only wait and consume_token, and the host launches the two parts on different streams. The paper covers 13 optimizations, evaluates on H800 and MI308X clusters from 8 to 64 GPUs for most kernels (128 GPUs for AllToAll; L20 GPUs for the low-latency AllGather), and reports 1.09x–44.97x speedups over PyTorch with NCCL or RCCL.

## What I found strong

First, I think the one-sided, signal-based decomposition of collectives is the right design. NCCL and RCCL synchronize before and after a collective, which limits overlap to the operator level. By breaking AllGather and ReduceScatter into one-sided point-to-point operations synchronized through signals, the authors get tile-granular overlap: each GEMM tile waits only for its own signal instead of for the whole collective. The push-versus-pull discussion for intra-node AllGather — push mode drops one synchronization but loses control of arrival order — is exactly the honest trade-off I want to see.

Second, I was impressed by the topology-aware swizzling. On NVIDIA's NVSwitch topology (200 GB/s uni-directional per link) each rank gathers from one peer at a time, while on AMD's full-mesh MI308X topology (50 GB/s per link, 350 GB/s aggregate) each rank gathers sub-chunks from all peers at every step. For inter-node GEMM ReduceScatter, the starting tile is shifted so rank 0 begins computing the data rank 5 needs.

Third, the distributed autotuner is a genuine compiler contribution. It cannot just re-run the target kernel: it has to reset all signals before every profile run, execute the whole target function (communication, computation, and host launch) as a unit, and then globally synchronize to aggregate tuning results from all devices into one unified best configuration. I also liked the resource-partition analysis. With ~170 GB/s NVLink against ~45 GB/s InfiniBand, the paper derives that a reduction operator faster than 470 GB/s achieves perfect overlap, so no more than 15 SMs are needed for reduction.

## My main concerns

1. The headline number, 44.97x on intra-node AllGather MoE, rests on a baseline the authors themselves describe as weak: the PyTorch+NCCL baseline for MoE uses Python loops for GroupGEMMs. The honest comparison is the GEMM kernels against FLUX, where Triton-distributed achieves 1.09x on intra-node AllGather GEMM and 95.60% of FLUX's performance on inter-node AllGather GEMM, and 96.36% on inter-node GEMM ReduceScatter — solid results, but a very different story from 44.97x. The abstract and Figure 1 should lead with the strong-baseline numbers and label the MoE numbers as against a naive baseline.

2. The inter-node FLUX comparisons are not measured on the same hardware. For both inter-node AllGather GEMM and inter-node GEMM ReduceScatter, the paper uses FLUX's performance as reported in the FLUX paper (for AllGather GEMM it also notes some problem-size data was not available). I cannot tell how much of the 3.6–4.4% gap to FLUX is real and how much is measurement setup. These two kernels should be re-measured against FLUX on the same H800 cluster, or the comparison clearly marked as indicative rather than head-to-head.

3. There is no end-to-end evaluation. Every result is at the kernel level: AG+GEMM, GEMM+RS, MoE kernels, flash decoding, AllToAll. The paper motivates itself with LLM training and inference at cluster scale, and cites ByteDance's COMET saving millions of GPU hours through overlap, but I never see a full model — not one training step, not one inference serving run, not even a full transformer block compiled through the pipeline. Kernel speedups do not always compose into model speedups. At least one end-to-end result, even a small one, would show whether the overlap survives contact with a real workload.

4. The low-latency AllGather numbers are estimates, not measurements. The paper estimates 13.5 microseconds for the low-latency implementation against about 25 microseconds for the baseline, and the skew analysis (a 0.5 microsecond NVLink transfer stretching to 1.5 microseconds) is also an estimate. The LL protocol doubles message size because flags travel with the data, which the paper says makes it suitable for small messages only — but I found no measurement of where that crossover actually is. Given that low-latency AllGather underpins the distributed flash decoding results, I would like measured latency and a stated message-size limit.

5. I found no discussion of correctness, debugging, or determinism for signal-based one-sided programs. Hand-written signal exchanges invite deadlocks, races, and stale-signal bugs that are hard to reproduce at 64-GPU scale. The paper explains how wait and consume_token create a data dependency for pipelining, but I found no correctness argument, no verification that the generated kernels match a reference implementation, and no tooling for diagnosing a hung or mis-synchronized kernel. For a compiler whose pitch is that ordinary Python developers can now write distributed kernels, I think this gap matters as much as the performance numbers.

## Smaller points and questions

- The AllToAll results cut both ways and the paper is honest about it: the Python implementation averages 1.18x on dispatch and 1.44x on combine over 8–64 GPUs, but loses on dispatch at 64 GPUs, and at 128 GPUs DeepEP is still best because it uses IBGDA while Triton-distributed uses IBRC, and the NVSHMEM bitcode library does not support IBGDA. Is there any path to IBGDA support? It caps the scalability story.

- The AMD evaluation is thin: only two kernels (AG+GEMM at 1.09x, GEMM+RS at 1.16x over PyTorch+RCCL), single-node only, and the Triton GEMM is slightly below rocBLAS. The paper describes a workaround on AMD — driver signal APIs like hipStreamWriteValue seem to interfere with compute kernels, so signals are sent by launching a separate data transfer — but I would like to see whether that workaround costs measurable bandwidth.

- The MoE ReduceScatter scaling result is a loose end: scaling from one node to two increases latency non-linearly, and the paper says a dedicated ReduceScatter kernel is needed and leaves it for future work. The MoE story is therefore incomplete on the inter-node side.

- The claim that the approach ports to NPUs is speculative. The paper says other platforms need symmetric memory, signal exchange, and async-tasks, and that OpenSHMEM support "can be expected" to be widely adopted — but no NPU hardware is tested. I would want at least a statement of which current NPUs meet those three requirements.

- The productivity claim — hundreds of lines of Python versus thousands of lines of CUDA for AllToAll — is stated without line counts or any development-time measurement. I believe the direction of the claim, but I would like the actual numbers.

## Overall assessment

I think this is a strong systems paper, and I would recommend it to anyone building compilers for distributed ML. Its contribution is not a single new algorithm but a coherent, usable system: OpenSHMEM primitives inside Triton, a clear MPMD programming model with symmetric memory and signals, topology-aware swizzling for two vendors, and a distributed autotuner that handles the synchronization problems honestly. The AllToAll result against DeepEP — a hand-written CUDA library — with a few hundred lines of Python is the single most convincing datapoint in the paper, and the authors report the cases where they lose, at 64 and 128 GPUs. My main reservation is the evaluation framing: the headline 44.97x rests on a weak baseline, the inter-node FLUX numbers are borrowed from another paper, and there is no end-to-end model result and no correctness story for hand-written signal programs. I hope the authors tighten those points, because the system itself deserves a stronger evidentiary case.

Competing interests

The author declares that they have no competing interests.

Use of Artificial Intelligence (AI)

The author declares that they used generative AI to come up with new ideas for their review.

Puedes escribir un comentario en esta PREreview de Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler.

Antes de comenzar

We will ask you to log in with your ORCID iD. If you don’t have an iD, you can create one.

What is an ORCID iD?

An ORCID iD is a unique identifier that distinguishes you from everyone with the same or similar name.

Comenzar ahora