| Issue |
244224
|
| Summary |
[AMDGPU] Vector fptrunc to bfloat is not packed when the result is bitcast to <N x i16> Labels: backend:AMDGPU, missed-optimization
|
| Labels |
new issue
|
| Assignees |
|
| Reporter |
yingluAMD
|
`fptrunc <N x float> to <N x bfloat>` (N >= 4) is split into scalar conversions during legalization. If the result is used as bfloat, isel pairs them back into `v_cvt_pk_bf16_f32`. If the result is bitcast to `<N x i16>`, a DAG combine on the bitcast breaks that pairing and every element gets its own half-used cvt. The bitcast itself is free; the problem is that vector bf16 `fptrunc` depends on that pairing to come out packed.
```llvm
; RUN: llc -mtriple=amdgpu9.50-amd-amdhsa < %s
; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa < %s
;
; The fptrunc is split into four scalar conversions. In @E, where the result
; stays <4 x bfloat>, isel pairs them back into two v_cvt_pk_bf16_f32. In @D
; the result is then bitcast to <4 x i16>, the pairing is lost, and it gets
; four half-used cvts plus v_perm_b32 (gfx950) / v_mov_b16 (gfx1250) to
; re-pack them.
define <4 x i16> @D(<4 x float> %a) {
%x = fptrunc <4 x float> %a to <4 x bfloat>
%y = bitcast <4 x bfloat> %x to <4 x i16>
ret <4 x i16> %y
}
define <4 x bfloat> @E(<4 x float> %a) {
%x = fptrunc <4 x float> %a to <4 x bfloat>
ret <4 x bfloat> %x
}
```
gfx950, `@D`:
```asm
v_cvt_pk_bf16_f32 v2, v2, s0
v_cvt_pk_bf16_f32 v3, v3, s0
v_cvt_pk_bf16_f32 v0, v0, s0
v_cvt_pk_bf16_f32 v1, v1, s0
s_mov_b32 s0, 0x5040100
v_perm_b32 v0, v1, v0, s0
v_perm_b32 v1, v3, v2, s0
```
gfx1250, `@D`:
```asm
v_cvt_pk_bf16_f32 v4, v1, s0
v_cvt_pk_bf16_f32 v3, v3, s0
v_cvt_pk_bf16_f32 v1, v2, s0
v_cvt_pk_bf16_f32 v0, v0, s0
v_mov_b16_e32 v0.h, v4.l
v_mov_b16_e32 v1.h, v3.l
```
Expected: the same code as `@E` on both targets, since the bitcast does not change the bits:
```asm
v_cvt_pk_bf16_f32 v0, v0, v1
v_cvt_pk_bf16_f32 v1, v2, v3
```
The `<N x i16>` form is what the gfx90a-era bf16 MFMAs take (`llvm.amdgcn.mfma.f32.*bf16.1k` have `<4 x i16>` operands), so feeding them `fptrunc` results always hits this. With two `<4 x float>` operands, gfx950 emits 8 cvt + 4 perm instead of 4 cvt:
```llvm
; RUN: llc -mtriple=amdgpu9.50-amd-amdhsa < %s
define <4 x float> @mfma_bad(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
%x = fptrunc <4 x float> %a to <4 x bfloat>
%y = fptrunc <4 x float> %b to <4 x bfloat>
%xi = bitcast <4 x bfloat> %x to <4 x i16>
%yi = bitcast <4 x bfloat> %y to <4 x i16>
%r = call <4 x float> @llvm.amdgcn.mfma.f32.16x16x16bf16.1k(<4 x i16> %xi, <4 x i16> %yi, <4 x float> %c, i32 0, i32 0, i32 0)
ret <4 x float> %r
}
```
>From HIP this is `__builtin_bit_cast(short4, __builtin_convertvector(v, bf16x4))` passed to `__builtin_amdgcn_mfma_f32_16x16x16bf16_1k`. (If the `short4` is only stored, InstCombine folds the bitcast into the store and the problem does not show.)
### Scope
`v_cvt_pk_bf16_f32` / `v_perm_b32` (gfx950) or `v_mov_b16` (gfx1250) counts; ideal is N/2 cvt and no perm/mov:
| IR | gfx950 | gfx1250 |
|---|---|---|
| `fptrunc` to `<N x bfloat>`, N = 2..16, used as bfloat | N/2 / 0 | N/2 / 0 |
| ... bitcast to `<N/2 x i32>` or `i64` | N/2 / 0 | N/2 / 0 |
| ... bitcast to `<4 x i16>` / `<8 x i16>` / `<16 x i16>` | 4/2, 8/4, 16/8 | 4/2, 8/4, 16/8 |
| `<3 x float>` to `<3 x bfloat>` to `<3 x i16>` | 3 / 1 | 3 / 1 |
| `<4 x double>` to `<4 x bfloat>` to `<4 x i16>` | 4 / 2 | 4 / 2 |
gfx1310 behaves like gfx950. The f16 equivalent (`fptrunc` to `<N x half>`, bitcast to `<N x i16>`) is packed on both targets.
### What happens
Excerpts (trimmed) from `llc -mtriple=amdgpu9.50-amd-amdhsa -debug-_only_=legalizevectorops,dagcombine,isel` on the IR above, with an assertions build. gfx1250 gives the same output, node numbers included, up to instruction selection.
1. **Vector op legalization scalarizes the `fp_round`.** Both `@D` and `@E` show:
```
Legalizing vector op: t11: v4bf16 = fp_round t9, TargetConstant:i64<0>
Expanding
```
`FP_ROUND` is Custom only for `bf16` and `v2bf16` ([SIISelLowering.cpp](https://github.com/llvm/llvm-project/blob/05b37c72e47db02a939728477b60a13f66622e08/llvm/lib/Target/AMDGPU/SIISelLowering.cpp#L1102-L1106)), so `v4bf16` gets the default expansion: four scalar `bf16 = fp_round` and a `v4bf16 BUILD_VECTOR`. From here on, a packed cvt only comes out if isel pairs the scalars up again.
2. **Generic combines narrow the halves** to `v2bf16 = BUILD_VECTOR (fp_round a), (fp_round b)`. In `@E` this is exactly what the pair pattern ([VOP3Instructions.td](https://github.com/llvm/llvm-project/blob/05b37c72e47db02a939728477b60a13f66622e08/llvm/lib/Target/AMDGPU/VOP3Instructions.td#L1864-L1866)) matches:
```
Morphed node: t29: v2bf16 = V_CVT_PK_BF16_F32_e64 0, t2, 0, t4, ...
```
3. **In `@D` the bitcast combine breaks the pair.** The `v2bf16` halves feed a `bitcast` to `v2i16`, and "Push casts through vector builds" in `AMDGPUTargetLowering::PerformDAGCombine` ([AMDGPUISelLowering.cpp](https://github.com/llvm/llvm-project/blob/05b37c72e47db02a939728477b60a13f66622e08/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp#L5578-L5607)) rewrites it:
```
Combining: t24: v2i16 = bitcast t34
... into: t37: v2i16 = BUILD_VECTOR t35, t36 ; t35, t36 = i16 bitcast of the bf16 fp_rounds
```
It fires because the result is a vector with the same element count as the `BUILD_VECTOR` and the DAG is not yet legalized. The element count check is why `i64` and `<2 x i32>` are fine. Its comment says it is meant for materializing floating point vector constants, but nothing restricts it to constant operands (the generic `DAGCombiner::visitBITCAST` fold only handles constant `BUILD_VECTOR`s).
4. **Isel selects each element alone.** The pair pattern needs a `v2bf16` `build_vector` of `fp_round`s and now sees a `v2i16` `build_vector` of `i16` bitcasts, so each `fp_round` matches the single-element pattern with an `IMPLICIT_DEF` second source (the `s0` in the asm above):
```
Morphed node: t32: bf16 = V_CVT_PK_BF16_F32_e64 0, t8, 0, IMPLICIT_DEF:f32, ...
```
On gfx950 the `v2i16 BUILD_VECTOR` becomes `V_PERM_B32` (selector `0x5040100`). On gfx1250 (true16) each cvt is followed by an `EXTRACT_SUBREG` of lo16, the `BUILD_VECTOR` becomes a lo16/hi16 `REG_SEQUENCE`, and after register allocation the hi16 half is a 16-bit `COPY` that ends up as `v_mov_b16`.
Step 1 is where the packed form is lost; steps 3 and 4 only decide whether isel can recover it.
The same scalarization existed for f16 and was fixed in #141883 by making `FP_ROUND` Custom for `v2f16..v16f16` and splitting into `v2` pieces in `lowerFP_ROUND` (`splitFP_ROUNDVectorOp`). Doing the same for `v4bf16/v8bf16/v16bf16` with an f32 source looks like the natural fix (not tried yet; f64 sources and `v3bf16` would need a look, since `splitFP_ROUNDVectorOp` asserts a power-of-two element count). That would make the result independent of the isel pairing, so the bitcast combine would not need to change.
This is not a regression: release 20.1.8, 21.1.8 and 22.1.0 (with `-mtriple=amdgcn-amd-amdhsa -mcpu=gfx950`) give the same 4 cvt + 2 perm for `@D`, i.e. it has been there since gfx950 support was added (#116678).
We hit this in a convolution library on gfx950 whose kernels feed fp32 data into the `_1k` MFMAs: the affected kernels execute twice the needed conversions, about +3% instructions.
### Version
LLVM 24.0.0git, llvm/llvm-project main at 05b37c72e4 (2026-11-13), built locally with assertions. Also reproduces with AMD clang 24.0.0git from ROCm 10.1.0 (ROCm/llvm-project d6f6cb6918) for the HIP form.
This issue was investigated and drafted with the help of an AI assistant (Claude). I have reviewed it and reproduced the results above.
_______________________________________________
llvm-bugs mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs