Issue 243335
Summary [AtomicExpand] Partword RMW cmpxchg loop reuses the initial load when the address is known aligned
Labels new issue
Assignees
Reporter Mramaran
    When `AtomicExpand` widens a sub-word `atomicrmw` into a 32-bit cmpxchg loop and the pointer is known to be 4-aligned (but the `atomicrmw` itself says `align 2`), the value extracted inside the loop comes from the load before the loop instead of from the loop phi. Every retry computes `initial_value op v`, so updates under contention are lost.

Reproducer (`opt -S -mtriple=nvptx64-nvidia-cuda -mcpu=sm_80 -passes='require<libcall-lowering-info>,atomic-expand'`):

```llvm
define bfloat @f(ptr addrspace(1) align 16 %p, bfloat %v) {
  %old = atomicrmw fadd ptr addrspace(1) %p, bfloat %v monotonic, align 2
  ret bfloat %old
}
```

Output (trimmed):

```llvm
  %3 = load atomic i32, ptr addrspace(1) %p monotonic, align 4
  br label %atomicrmw.start

atomicrmw.start:
  %loaded = phi i32 [ %3, %0 ], [ %newloaded, %atomicrmw.start ]
  %extracted = trunc i32 %3 to i16          ; should use %loaded
  %4 = bitcast i16 %extracted to bfloat
  %new = fadd bfloat %4, %v
  ...
  %6 = cmpxchg ptr addrspace(1) %p, i32 %loaded, i32 %inserted monotonic monotonic, align 4
```

Without `align 16` on `%p` the loop is correct (`%shifted = lshr i32 %loaded, %ShiftAmt`). With `align 4` on the `atomicrmw` it is also correct.

Cause: `insertRMWCmpXchgLoop` creates `%loaded` with only the entry incoming value and then calls `PerformOp(Builder, Loaded)` before the back edge is added. `ReplacementIRBuilder` uses `InstSimplifyFolder`. Since `%p` is known 4-aligned, `ShiftAmt` is known zero, and simplifying `lshr %loaded, %ShiftAmt` threads through the one-input phi and returns `%3`. The back edge added afterwards doesn't undo the fold.

Affected: any partword RMW whose `PerformOp` extracts the narrow value inside the loop. On NVPTX I see it for `fadd`/`fsub`/`fmax` on `bfloat`, `fadd half` on sm_60, and `umax i16`. `add`, `xchg` and `nand` on `i16` work on the wide word and are fine.

Impact: Triton hit this on sm_80/sm_89 for `tl.atomic_add` on bf16 (triton-lang/triton#12740). 128 threads adding 1.0 to the same address produce 1.0.

Possible fixes: build the in-loop ops without the simplifying folder, or create the phi with both incoming values before running `PerformOp` (for example with a placeholder for the back edge that is replaced afterwards).

Seen at llvm-project 18f9e623e4c8. The same code is on `main`.

_______________________________________________
llvm-bugs mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs

Reply via email to