[llvm-branch-commits] [llvm] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from VOP3 (PR #196516)

2026-07-16 Thread via llvm-branch-commits 

llvmorg-github-actions[bot] wrote:




@llvm/pr-subscribers-backend-amdgpu

Author: Petar Avramovic (petar-avramovic)


Changes

For V_DOT2_F32_F16 and V_DOT2_F32_BF16 add their VOPDName and mark
them with usesCustomInserter which will be used to add pre-RA register
allocation hints to preferably assign dst and src2 to the same physical
register. When the hint is satisfied, canMapVOP3PToVOPD recognises the
instruction as eligible for VOPD pairing by checking if it is VOP2 like:
dst==src2, no source modifiers, no clamp, and src1 is a register.
Mark both instructions as commutable to allow a literal in src1 to be
moved to src0, since VOPD only permits a literal in src0.

Original patch had a bug where it did not check if physical src
registers match register class of appropriate operand in fullVOPD
instructions, check is now done via isValidVOPDSrc.

---

Patch is 87.92 KiB, truncated to 20.00 KiB below, full version: 
https://github.com/llvm/llvm-project/pull/196516.diff


7 Files Affected:

- (modified) llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp (+34-1) 
- (modified) llvm/lib/Target/AMDGPU/SIISelLowering.cpp (+8) 
- (modified) llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp (+6) 
- (modified) llvm/lib/Target/AMDGPU/VOP3PInstructions.td (+8-5) 
- (modified) llvm/lib/Target/AMDGPU/VOPInstructions.td (+2-2) 
- (modified) llvm/test/CodeGen/AMDGPU/llvm.amdgcn.fdot2.f32.bf16.ll (+163-69) 
- (modified) llvm/test/CodeGen/AMDGPU/llvm.amdgcn.fdot2.ll (+447-525) 


``diff
diff --git a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp 
b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
index 0e1d3693b487c..54d1910a7c05d 100644
--- a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
@@ -56,6 +56,38 @@ bool isValidVOPDSrc(const SIInstrInfo &TII, int VOPDOpc, 
unsigned CompIdx,
   return TII.getRegClass(TII.get(VOPDOpc), OpIdx)->contains(PhysSrcReg);
 }
 
+static const MachineOperand &getNamedOp(const MachineInstr &MI,
+AMDGPU::OpName Name) {
+  return MI.getOperand(getNamedOperandIdx(MI.getOpcode(), Name));
+}
+
+// Check if MI is a VOP3P instruction with operands that satisfy the 
constraints
+// for mapping it to a VOP2/VOPD opcode: no modifiers, no clamp, src1 and src2
+// are registers (src0 can be register or literal), and src2 is same as dst.
+static bool canMapVOP3PToVOPD(const MachineInstr &MI) {
+  unsigned Opc = MI.getOpcode();
+  if (Opc != AMDGPU::V_DOT2_F32_F16 && Opc != AMDGPU::V_DOT2_F32_BF16)
+return false;
+  // src0 can be register or literal
+  if (getNamedOp(MI, AMDGPU::OpName::src0_modifiers).getImm() !=
+  SISrcMods::OP_SEL_1)
+return false;
+  if (getNamedOp(MI, AMDGPU::OpName::src1_modifiers).getImm() !=
+  SISrcMods::OP_SEL_1)
+return false;
+  if (!getNamedOp(MI, AMDGPU::OpName::src1).isReg())
+return false;
+  if (getNamedOp(MI, AMDGPU::OpName::src2_modifiers).getImm() !=
+  SISrcMods::OP_SEL_1)
+return false;
+  if (!getNamedOp(MI, AMDGPU::OpName::src2).isReg())
+return false;
+  if (getNamedOp(MI, AMDGPU::OpName::clamp).getImm() != 0)
+return false;
+  return getNamedOp(MI, AMDGPU::OpName::vdst).getReg() ==
+ getNamedOp(MI, AMDGPU::OpName::src2).getReg();
+}
+
 bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
const MachineInstr &MIX,
const MachineInstr &MIY, bool IsVOPD3,
@@ -67,7 +99,8 @@ bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
 
   if (IsVOPD3 && !ST.hasVOPD3())
 return false;
-  if (!IsVOPD3 && (TII.isVOP3(MIX) || TII.isVOP3(MIY)))
+  if (!IsVOPD3 && ((TII.isVOP3(MIX) && !canMapVOP3PToVOPD(MIX)) ||
+   (TII.isVOP3(MIY) && !canMapVOP3PToVOPD(MIY
 return false;
   if (TII.isDPP(MIX) || TII.isDPP(MIY))
 return false;
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp 
b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index 8f406f2d8ef74..9ab55ea177c16 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -7351,6 +7351,14 @@ 
SITargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
 MI.getOperand(0).setReg(OriginalExec);
 return BB;
   }
+  case AMDGPU::V_DOT2_F32_F16:
+  case AMDGPU::V_DOT2_F32_BF16: {
+// Hint RA to assign dst and src2 the same physical register.
+// For targets without VOP2, but with VOPD, variant of the instruction this
+// is one of the conditions to attempt converting VOP3P to VOPD.
+MRI.setSimpleHint(MI.getOperand(0).getReg(), MI.getOperand(6).getReg());
+return BB;
+  }
   default:
 if (TII->isImage(MI) || TII->isMUBUF(MI)) {
   if (!MI.mayStore())
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp 
b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
index 71d95a23e30d3..99b408e0cc35c 100644
--- a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
@@ -928,6 +928,12 @@ ComponentPr

[llvm-branch-commits] [llvm] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from VOP3 (PR #196516)

2026-07-16 Thread Petar Avramovic via llvm-branch-commits 

https://github.com/petar-avramovic ready_for_review 
https://github.com/llvm/llvm-project/pull/196516
___
llvm-branch-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

[llvm-branch-commits] [llvm] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from VOP3 (PR #196516)

2026-07-16 Thread Petar Avramovic via llvm-branch-commits 

https://github.com/petar-avramovic updated 
https://github.com/llvm/llvm-project/pull/196516

>From 9a839d52b357d50fd21301cb5eb71426ef23bbdd Mon Sep 17 00:00:00 2001
From: Petar Avramovic 
Date: Fri, 8 May 2026 11:07:09 +0200
Subject: [PATCH] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from
 VOP3

For V_DOT2_F32_F16 and V_DOT2_F32_BF16 add their VOPDName and mark
them with usesCustomInserter which will be used to add pre-RA register
allocation hints to preferably assign dst and src2 to the same physical
register. When the hint is satisfied, canMapVOP3PToVOPD recognises the
instruction as eligible for VOPD pairing by checking if it is VOP2 like:
dst==src2, no source modifiers, no clamp, and src1 is a register.
Mark both instructions as commutable to allow a literal in src1 to be
moved to src0, since VOPD only permits a literal in src0.

Original patch had a bug where it did not check if physical src
registers match register class of appropriate operand in fullVOPD
instructions, check is now done via isValidVOPDSrc.
---
 llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp   |  35 +-
 llvm/lib/Target/AMDGPU/SIISelLowering.cpp |   8 +
 .../Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp|   6 +
 llvm/lib/Target/AMDGPU/VOP3PInstructions.td   |  13 +-
 llvm/lib/Target/AMDGPU/VOPInstructions.td |   4 +-
 .../AMDGPU/llvm.amdgcn.fdot2.f32.bf16.ll  | 232 +++--
 llvm/test/CodeGen/AMDGPU/llvm.amdgcn.fdot2.ll | 972 --
 7 files changed, 668 insertions(+), 602 deletions(-)

diff --git a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp 
b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
index 0e1d3693b487c..54d1910a7c05d 100644
--- a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
@@ -56,6 +56,38 @@ bool isValidVOPDSrc(const SIInstrInfo &TII, int VOPDOpc, 
unsigned CompIdx,
   return TII.getRegClass(TII.get(VOPDOpc), OpIdx)->contains(PhysSrcReg);
 }
 
+static const MachineOperand &getNamedOp(const MachineInstr &MI,
+AMDGPU::OpName Name) {
+  return MI.getOperand(getNamedOperandIdx(MI.getOpcode(), Name));
+}
+
+// Check if MI is a VOP3P instruction with operands that satisfy the 
constraints
+// for mapping it to a VOP2/VOPD opcode: no modifiers, no clamp, src1 and src2
+// are registers (src0 can be register or literal), and src2 is same as dst.
+static bool canMapVOP3PToVOPD(const MachineInstr &MI) {
+  unsigned Opc = MI.getOpcode();
+  if (Opc != AMDGPU::V_DOT2_F32_F16 && Opc != AMDGPU::V_DOT2_F32_BF16)
+return false;
+  // src0 can be register or literal
+  if (getNamedOp(MI, AMDGPU::OpName::src0_modifiers).getImm() !=
+  SISrcMods::OP_SEL_1)
+return false;
+  if (getNamedOp(MI, AMDGPU::OpName::src1_modifiers).getImm() !=
+  SISrcMods::OP_SEL_1)
+return false;
+  if (!getNamedOp(MI, AMDGPU::OpName::src1).isReg())
+return false;
+  if (getNamedOp(MI, AMDGPU::OpName::src2_modifiers).getImm() !=
+  SISrcMods::OP_SEL_1)
+return false;
+  if (!getNamedOp(MI, AMDGPU::OpName::src2).isReg())
+return false;
+  if (getNamedOp(MI, AMDGPU::OpName::clamp).getImm() != 0)
+return false;
+  return getNamedOp(MI, AMDGPU::OpName::vdst).getReg() ==
+ getNamedOp(MI, AMDGPU::OpName::src2).getReg();
+}
+
 bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
const MachineInstr &MIX,
const MachineInstr &MIY, bool IsVOPD3,
@@ -67,7 +99,8 @@ bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
 
   if (IsVOPD3 && !ST.hasVOPD3())
 return false;
-  if (!IsVOPD3 && (TII.isVOP3(MIX) || TII.isVOP3(MIY)))
+  if (!IsVOPD3 && ((TII.isVOP3(MIX) && !canMapVOP3PToVOPD(MIX)) ||
+   (TII.isVOP3(MIY) && !canMapVOP3PToVOPD(MIY
 return false;
   if (TII.isDPP(MIX) || TII.isDPP(MIY))
 return false;
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp 
b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index 8f406f2d8ef74..9ab55ea177c16 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -7351,6 +7351,14 @@ 
SITargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
 MI.getOperand(0).setReg(OriginalExec);
 return BB;
   }
+  case AMDGPU::V_DOT2_F32_F16:
+  case AMDGPU::V_DOT2_F32_BF16: {
+// Hint RA to assign dst and src2 the same physical register.
+// For targets without VOP2, but with VOPD, variant of the instruction this
+// is one of the conditions to attempt converting VOP3P to VOPD.
+MRI.setSimpleHint(MI.getOperand(0).getReg(), MI.getOperand(6).getReg());
+return BB;
+  }
   default:
 if (TII->isImage(MI) || TII->isMUBUF(MI)) {
   if (!MI.mayStore())
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp 
b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
index 71d95a23e30d3..99b408e0cc35c 100644
--- a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
@@ -928,6 +92

[llvm-branch-commits] [llvm] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from VOP3 (PR #196516)

2026-07-16 Thread Petar Avramovic via llvm-branch-commits 

https://github.com/petar-avramovic updated 
https://github.com/llvm/llvm-project/pull/196516

>From 9a839d52b357d50fd21301cb5eb71426ef23bbdd Mon Sep 17 00:00:00 2001
From: Petar Avramovic 
Date: Fri, 8 May 2026 11:07:09 +0200
Subject: [PATCH] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from
 VOP3

For V_DOT2_F32_F16 and V_DOT2_F32_BF16 add their VOPDName and mark
them with usesCustomInserter which will be used to add pre-RA register
allocation hints to preferably assign dst and src2 to the same physical
register. When the hint is satisfied, canMapVOP3PToVOPD recognises the
instruction as eligible for VOPD pairing by checking if it is VOP2 like:
dst==src2, no source modifiers, no clamp, and src1 is a register.
Mark both instructions as commutable to allow a literal in src1 to be
moved to src0, since VOPD only permits a literal in src0.

Original patch had a bug where it did not check if physical src
registers match register class of appropriate operand in fullVOPD
instructions, check is now done via isValidVOPDSrc.
---
 llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp   |  35 +-
 llvm/lib/Target/AMDGPU/SIISelLowering.cpp |   8 +
 .../Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp|   6 +
 llvm/lib/Target/AMDGPU/VOP3PInstructions.td   |  13 +-
 llvm/lib/Target/AMDGPU/VOPInstructions.td |   4 +-
 .../AMDGPU/llvm.amdgcn.fdot2.f32.bf16.ll  | 232 +++--
 llvm/test/CodeGen/AMDGPU/llvm.amdgcn.fdot2.ll | 972 --
 7 files changed, 668 insertions(+), 602 deletions(-)

diff --git a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp 
b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
index 0e1d3693b487c..54d1910a7c05d 100644
--- a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
@@ -56,6 +56,38 @@ bool isValidVOPDSrc(const SIInstrInfo &TII, int VOPDOpc, 
unsigned CompIdx,
   return TII.getRegClass(TII.get(VOPDOpc), OpIdx)->contains(PhysSrcReg);
 }
 
+static const MachineOperand &getNamedOp(const MachineInstr &MI,
+AMDGPU::OpName Name) {
+  return MI.getOperand(getNamedOperandIdx(MI.getOpcode(), Name));
+}
+
+// Check if MI is a VOP3P instruction with operands that satisfy the 
constraints
+// for mapping it to a VOP2/VOPD opcode: no modifiers, no clamp, src1 and src2
+// are registers (src0 can be register or literal), and src2 is same as dst.
+static bool canMapVOP3PToVOPD(const MachineInstr &MI) {
+  unsigned Opc = MI.getOpcode();
+  if (Opc != AMDGPU::V_DOT2_F32_F16 && Opc != AMDGPU::V_DOT2_F32_BF16)
+return false;
+  // src0 can be register or literal
+  if (getNamedOp(MI, AMDGPU::OpName::src0_modifiers).getImm() !=
+  SISrcMods::OP_SEL_1)
+return false;
+  if (getNamedOp(MI, AMDGPU::OpName::src1_modifiers).getImm() !=
+  SISrcMods::OP_SEL_1)
+return false;
+  if (!getNamedOp(MI, AMDGPU::OpName::src1).isReg())
+return false;
+  if (getNamedOp(MI, AMDGPU::OpName::src2_modifiers).getImm() !=
+  SISrcMods::OP_SEL_1)
+return false;
+  if (!getNamedOp(MI, AMDGPU::OpName::src2).isReg())
+return false;
+  if (getNamedOp(MI, AMDGPU::OpName::clamp).getImm() != 0)
+return false;
+  return getNamedOp(MI, AMDGPU::OpName::vdst).getReg() ==
+ getNamedOp(MI, AMDGPU::OpName::src2).getReg();
+}
+
 bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
const MachineInstr &MIX,
const MachineInstr &MIY, bool IsVOPD3,
@@ -67,7 +99,8 @@ bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
 
   if (IsVOPD3 && !ST.hasVOPD3())
 return false;
-  if (!IsVOPD3 && (TII.isVOP3(MIX) || TII.isVOP3(MIY)))
+  if (!IsVOPD3 && ((TII.isVOP3(MIX) && !canMapVOP3PToVOPD(MIX)) ||
+   (TII.isVOP3(MIY) && !canMapVOP3PToVOPD(MIY
 return false;
   if (TII.isDPP(MIX) || TII.isDPP(MIY))
 return false;
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp 
b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index 8f406f2d8ef74..9ab55ea177c16 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -7351,6 +7351,14 @@ 
SITargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
 MI.getOperand(0).setReg(OriginalExec);
 return BB;
   }
+  case AMDGPU::V_DOT2_F32_F16:
+  case AMDGPU::V_DOT2_F32_BF16: {
+// Hint RA to assign dst and src2 the same physical register.
+// For targets without VOP2, but with VOPD, variant of the instruction this
+// is one of the conditions to attempt converting VOP3P to VOPD.
+MRI.setSimpleHint(MI.getOperand(0).getReg(), MI.getOperand(6).getReg());
+return BB;
+  }
   default:
 if (TII->isImage(MI) || TII->isMUBUF(MI)) {
   if (!MI.mayStore())
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp 
b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
index 71d95a23e30d3..99b408e0cc35c 100644
--- a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
@@ -928,6 +92

[llvm-branch-commits] [llvm] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from VOP3 (PR #196516)

2026-06-16 Thread Petar Avramovic via llvm-branch-commits 

https://github.com/petar-avramovic updated 
https://github.com/llvm/llvm-project/pull/196516

>From b24bc4da58fa8b9819f28c18013a3bfd9e86746b Mon Sep 17 00:00:00 2001
From: Petar Avramovic 
Date: Fri, 8 May 2026 11:07:09 +0200
Subject: [PATCH] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from
 VOP3

For V_DOT2_F32_F16 and V_DOT2_F32_BF16 add their VOPDName and mark
them with usesCustomInserter which will be used to add pre-RA register
allocation hints to preferably assign dst and src2 to the same physical
register. When the hint is satisfied, canMapVOP3PToVOPD recognises the
instruction as eligible for VOPD pairing by checking if it is VOP2 like:
dst==src2, no source modifiers, no clamp, and src1 is a register.
Mark both instructions as commutable to allow a literal in src1 to be
moved to src0, since VOPD only permits a literal in src0.

Original patch had a bug where it did not check if physical src
registers match register class of appropriate operand in fullVOPD
instructions, check is now done via isValidVOPDSrc.
---
 llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp   |  35 +-
 llvm/lib/Target/AMDGPU/SIISelLowering.cpp |   8 +
 .../Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp|   6 +
 llvm/lib/Target/AMDGPU/VOP3PInstructions.td   |  13 +-
 llvm/lib/Target/AMDGPU/VOPInstructions.td |   4 +-
 .../AMDGPU/llvm.amdgcn.fdot2.f32.bf16.ll  | 232 +++--
 llvm/test/CodeGen/AMDGPU/llvm.amdgcn.fdot2.ll | 972 --
 7 files changed, 668 insertions(+), 602 deletions(-)

diff --git a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp 
b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
index 8297ba7e3425c..e6c9fb6d168d1 100644
--- a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
@@ -56,6 +56,38 @@ bool isValidVOPDSrc(const SIInstrInfo &TII, int VOPDOpc, 
unsigned CompIdx,
   return TII.getRegClass(TII.get(VOPDOpc), OpIdx)->contains(PhysSrcReg);
 }
 
+static const MachineOperand &getNamedOp(const MachineInstr &MI,
+AMDGPU::OpName Name) {
+  return MI.getOperand(getNamedOperandIdx(MI.getOpcode(), Name));
+}
+
+// Check if MI is a VOP3P instruction with operands that satisfy the 
constraints
+// for mapping it to a VOP2/VOPD opcode: no modifiers, no clamp, src1 and src2
+// are registers (src0 can be register or literal), and src2 is same as dst.
+static bool canMapVOP3PToVOPD(const MachineInstr &MI) {
+  unsigned Opc = MI.getOpcode();
+  if (Opc != AMDGPU::V_DOT2_F32_F16 && Opc != AMDGPU::V_DOT2_F32_BF16)
+return false;
+  // src0 can be register or literal
+  if (getNamedOp(MI, AMDGPU::OpName::src0_modifiers).getImm() !=
+  SISrcMods::OP_SEL_1)
+return false;
+  if (getNamedOp(MI, AMDGPU::OpName::src1_modifiers).getImm() !=
+  SISrcMods::OP_SEL_1)
+return false;
+  if (!getNamedOp(MI, AMDGPU::OpName::src1).isReg())
+return false;
+  if (getNamedOp(MI, AMDGPU::OpName::src2_modifiers).getImm() !=
+  SISrcMods::OP_SEL_1)
+return false;
+  if (!getNamedOp(MI, AMDGPU::OpName::src2).isReg())
+return false;
+  if (getNamedOp(MI, AMDGPU::OpName::clamp).getImm() != 0)
+return false;
+  return getNamedOp(MI, AMDGPU::OpName::vdst).getReg() ==
+ getNamedOp(MI, AMDGPU::OpName::src2).getReg();
+}
+
 bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
const MachineInstr &MIX,
const MachineInstr &MIY, bool IsVOPD3,
@@ -67,7 +99,8 @@ bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
 
   if (IsVOPD3 && !ST.hasVOPD3())
 return false;
-  if (!IsVOPD3 && (TII.isVOP3(MIX) || TII.isVOP3(MIY)))
+  if (!IsVOPD3 && ((TII.isVOP3(MIX) && !canMapVOP3PToVOPD(MIX)) ||
+   (TII.isVOP3(MIY) && !canMapVOP3PToVOPD(MIY
 return false;
   if (TII.isDPP(MIX) || TII.isDPP(MIY))
 return false;
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp 
b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index 2817f1ac8c3ad..c87a9dfb0603a 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -7319,6 +7319,14 @@ 
SITargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
 MI.getOperand(0).setReg(OriginalExec);
 return BB;
   }
+  case AMDGPU::V_DOT2_F32_F16:
+  case AMDGPU::V_DOT2_F32_BF16: {
+// Hint RA to assign dst and src2 the same physical register.
+// For targets without VOP2, but with VOPD, variant of the instruction this
+// is one of the conditions to attempt converting VOP3P to VOPD.
+MRI.setSimpleHint(MI.getOperand(0).getReg(), MI.getOperand(6).getReg());
+return BB;
+  }
   default:
 if (TII->isImage(MI) || TII->isMUBUF(MI)) {
   if (!MI.mayStore())
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp 
b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
index 7c283be411b20..1b2c7f368fd06 100644
--- a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
@@ -947,6 +94

[llvm-branch-commits] [llvm] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from VOP3 (PR #196516)

2026-06-16 Thread Petar Avramovic via llvm-branch-commits 

https://github.com/petar-avramovic updated 
https://github.com/llvm/llvm-project/pull/196516

>From b24bc4da58fa8b9819f28c18013a3bfd9e86746b Mon Sep 17 00:00:00 2001
From: Petar Avramovic 
Date: Fri, 8 May 2026 11:07:09 +0200
Subject: [PATCH] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from
 VOP3

For V_DOT2_F32_F16 and V_DOT2_F32_BF16 add their VOPDName and mark
them with usesCustomInserter which will be used to add pre-RA register
allocation hints to preferably assign dst and src2 to the same physical
register. When the hint is satisfied, canMapVOP3PToVOPD recognises the
instruction as eligible for VOPD pairing by checking if it is VOP2 like:
dst==src2, no source modifiers, no clamp, and src1 is a register.
Mark both instructions as commutable to allow a literal in src1 to be
moved to src0, since VOPD only permits a literal in src0.

Original patch had a bug where it did not check if physical src
registers match register class of appropriate operand in fullVOPD
instructions, check is now done via isValidVOPDSrc.
---
 llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp   |  35 +-
 llvm/lib/Target/AMDGPU/SIISelLowering.cpp |   8 +
 .../Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp|   6 +
 llvm/lib/Target/AMDGPU/VOP3PInstructions.td   |  13 +-
 llvm/lib/Target/AMDGPU/VOPInstructions.td |   4 +-
 .../AMDGPU/llvm.amdgcn.fdot2.f32.bf16.ll  | 232 +++--
 llvm/test/CodeGen/AMDGPU/llvm.amdgcn.fdot2.ll | 972 --
 7 files changed, 668 insertions(+), 602 deletions(-)

diff --git a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp 
b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
index 8297ba7e3425c..e6c9fb6d168d1 100644
--- a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
@@ -56,6 +56,38 @@ bool isValidVOPDSrc(const SIInstrInfo &TII, int VOPDOpc, 
unsigned CompIdx,
   return TII.getRegClass(TII.get(VOPDOpc), OpIdx)->contains(PhysSrcReg);
 }
 
+static const MachineOperand &getNamedOp(const MachineInstr &MI,
+AMDGPU::OpName Name) {
+  return MI.getOperand(getNamedOperandIdx(MI.getOpcode(), Name));
+}
+
+// Check if MI is a VOP3P instruction with operands that satisfy the 
constraints
+// for mapping it to a VOP2/VOPD opcode: no modifiers, no clamp, src1 and src2
+// are registers (src0 can be register or literal), and src2 is same as dst.
+static bool canMapVOP3PToVOPD(const MachineInstr &MI) {
+  unsigned Opc = MI.getOpcode();
+  if (Opc != AMDGPU::V_DOT2_F32_F16 && Opc != AMDGPU::V_DOT2_F32_BF16)
+return false;
+  // src0 can be register or literal
+  if (getNamedOp(MI, AMDGPU::OpName::src0_modifiers).getImm() !=
+  SISrcMods::OP_SEL_1)
+return false;
+  if (getNamedOp(MI, AMDGPU::OpName::src1_modifiers).getImm() !=
+  SISrcMods::OP_SEL_1)
+return false;
+  if (!getNamedOp(MI, AMDGPU::OpName::src1).isReg())
+return false;
+  if (getNamedOp(MI, AMDGPU::OpName::src2_modifiers).getImm() !=
+  SISrcMods::OP_SEL_1)
+return false;
+  if (!getNamedOp(MI, AMDGPU::OpName::src2).isReg())
+return false;
+  if (getNamedOp(MI, AMDGPU::OpName::clamp).getImm() != 0)
+return false;
+  return getNamedOp(MI, AMDGPU::OpName::vdst).getReg() ==
+ getNamedOp(MI, AMDGPU::OpName::src2).getReg();
+}
+
 bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
const MachineInstr &MIX,
const MachineInstr &MIY, bool IsVOPD3,
@@ -67,7 +99,8 @@ bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
 
   if (IsVOPD3 && !ST.hasVOPD3())
 return false;
-  if (!IsVOPD3 && (TII.isVOP3(MIX) || TII.isVOP3(MIY)))
+  if (!IsVOPD3 && ((TII.isVOP3(MIX) && !canMapVOP3PToVOPD(MIX)) ||
+   (TII.isVOP3(MIY) && !canMapVOP3PToVOPD(MIY
 return false;
   if (TII.isDPP(MIX) || TII.isDPP(MIY))
 return false;
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp 
b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index 2817f1ac8c3ad..c87a9dfb0603a 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -7319,6 +7319,14 @@ 
SITargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
 MI.getOperand(0).setReg(OriginalExec);
 return BB;
   }
+  case AMDGPU::V_DOT2_F32_F16:
+  case AMDGPU::V_DOT2_F32_BF16: {
+// Hint RA to assign dst and src2 the same physical register.
+// For targets without VOP2, but with VOPD, variant of the instruction this
+// is one of the conditions to attempt converting VOP3P to VOPD.
+MRI.setSimpleHint(MI.getOperand(0).getReg(), MI.getOperand(6).getReg());
+return BB;
+  }
   default:
 if (TII->isImage(MI) || TII->isMUBUF(MI)) {
   if (!MI.mayStore())
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp 
b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
index 7c283be411b20..1b2c7f368fd06 100644
--- a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
@@ -947,6 +94

[llvm-branch-commits] [llvm] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from VOP3 (PR #196516)

2026-06-15 Thread Mirko Brkušanin via llvm-branch-commits 

https://github.com/mbrkusanin edited 
https://github.com/llvm/llvm-project/pull/196516
___
llvm-branch-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

[llvm-branch-commits] [llvm] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from VOP3 (PR #196516)

2026-06-10 Thread Petar Avramovic via llvm-branch-commits 

https://github.com/petar-avramovic updated 
https://github.com/llvm/llvm-project/pull/196516

>From 7ca6f53f91595ee0781de75667c039d2a170a4a6 Mon Sep 17 00:00:00 2001
From: Petar Avramovic 
Date: Fri, 8 May 2026 11:07:09 +0200
Subject: [PATCH] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from
 VOP3

For V_DOT2_F32_F16 and V_DOT2_F32_BF16 add their VOPDName and mark
them with usesCustomInserter which will be used to add pre-RA register
allocation hints to preferably assign dst and src2 to the same physical
register. When the hint is satisfied, canMapVOP3PToVOPD recognises the
instruction as eligible for VOPD pairing by checking if it is VOP2 like:
dst==src2, no source modifiers, no clamp, and src1 is a register.
Mark both instructions as commutable to allow a literal in src1 to be
moved to src0, since VOPD only permits a literal in src0.

Original patch had a bug where it did not check if physical src
registers match register class of appropriate operand in fullVOPD
instructions, check is now done via isValidVOPDSrc.
---
 llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp   |  33 +-
 llvm/lib/Target/AMDGPU/SIISelLowering.cpp |   8 +
 .../Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp|   6 +
 llvm/lib/Target/AMDGPU/VOP3PInstructions.td   |  13 +-
 llvm/lib/Target/AMDGPU/VOPInstructions.td |   4 +-
 .../AMDGPU/llvm.amdgcn.fdot2.f32.bf16.ll  | 235 +++--
 llvm/test/CodeGen/AMDGPU/llvm.amdgcn.fdot2.ll | 972 --
 7 files changed, 669 insertions(+), 602 deletions(-)

diff --git a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp 
b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
index da57b0725065f..94a26440cc7c8 100644
--- a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
@@ -56,6 +56,36 @@ bool isValidVOPDSrc(const SIInstrInfo &TII, int VOPDOpc, 
unsigned CompIdx,
   return TII.getRegClass(TII.get(VOPDOpc), OpIdx)->contains(PhysSrcReg);
 }
 
+// Check if MI is a VOP3P instruction with operands that satisfy the 
constraints
+// for mapping it to a VOP2/VOPD opcode: no modifiers, no clamp, src1 and src2
+// are registers (src0 can be register or literal), and src2 is same as dst.
+static bool canMapVOP3PToVOPD(const MachineInstr &MI) {
+  unsigned Opc = MI.getOpcode();
+  if (Opc != AMDGPU::V_DOT2_F32_F16 && Opc != AMDGPU::V_DOT2_F32_BF16)
+return false;
+  // src0 can be register or literal
+  int16_t Src0ModsIdx = getNamedOperandIdx(Opc, 
AMDGPU::OpName::src0_modifiers);
+  if (MI.getOperand(Src0ModsIdx).getImm() != SISrcMods::OP_SEL_1)
+return false;
+  int16_t Src1ModsIdx = getNamedOperandIdx(Opc, 
AMDGPU::OpName::src1_modifiers);
+  if (MI.getOperand(Src1ModsIdx).getImm() != SISrcMods::OP_SEL_1)
+return false;
+  int16_t Src1Idx = getNamedOperandIdx(Opc, AMDGPU::OpName::src1);
+  if (!MI.getOperand(Src1Idx).isReg())
+return false;
+  int16_t Src2ModsIdx = getNamedOperandIdx(Opc, 
AMDGPU::OpName::src2_modifiers);
+  if (MI.getOperand(Src2ModsIdx).getImm() != SISrcMods::OP_SEL_1)
+return false;
+  int16_t Src2Idx = getNamedOperandIdx(Opc, AMDGPU::OpName::src2);
+  if (!MI.getOperand(Src2Idx).isReg())
+return false;
+  int16_t ClampIdx = getNamedOperandIdx(Opc, AMDGPU::OpName::clamp);
+  if (MI.getOperand(ClampIdx).getImm() != 0)
+return false;
+  int16_t VdstIdx = getNamedOperandIdx(Opc, AMDGPU::OpName::vdst);
+  return MI.getOperand(VdstIdx).getReg() == MI.getOperand(Src2Idx).getReg();
+}
+
 bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
const MachineInstr &MIX,
const MachineInstr &MIY, bool IsVOPD3,
@@ -67,7 +97,8 @@ bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
 
   if (IsVOPD3 && !ST.hasVOPD3())
 return false;
-  if (!IsVOPD3 && (TII.isVOP3(MIX) || TII.isVOP3(MIY)))
+  if (!IsVOPD3 && ((TII.isVOP3(MIX) && !canMapVOP3PToVOPD(MIX)) ||
+   (TII.isVOP3(MIY) && !canMapVOP3PToVOPD(MIY
 return false;
   if (TII.isDPP(MIX) || TII.isDPP(MIY))
 return false;
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp 
b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index a6dc58e6da263..96918c9f6dd36 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -7297,6 +7297,14 @@ 
SITargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
 MI.getOperand(0).setReg(OriginalExec);
 return BB;
   }
+  case AMDGPU::V_DOT2_F32_F16:
+  case AMDGPU::V_DOT2_F32_BF16: {
+// Hint RA to assign dst and src2 the same physical register.
+// For targets without VOP2, but with VOPD, variant of the instruction this
+// is one of the conditions to attempt converting VOP3P to VOPD.
+MRI.setSimpleHint(MI.getOperand(0).getReg(), MI.getOperand(6).getReg());
+return BB;
+  }
   default:
 if (TII->isImage(MI) || TII->isMUBUF(MI)) {
   if (!MI.mayStore())
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp 
b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
in

[llvm-branch-commits] [llvm] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from VOP3 (PR #196516)

2026-06-10 Thread Petar Avramovic via llvm-branch-commits 

https://github.com/petar-avramovic updated 
https://github.com/llvm/llvm-project/pull/196516

>From 7ca6f53f91595ee0781de75667c039d2a170a4a6 Mon Sep 17 00:00:00 2001
From: Petar Avramovic 
Date: Fri, 8 May 2026 11:07:09 +0200
Subject: [PATCH] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from
 VOP3

For V_DOT2_F32_F16 and V_DOT2_F32_BF16 add their VOPDName and mark
them with usesCustomInserter which will be used to add pre-RA register
allocation hints to preferably assign dst and src2 to the same physical
register. When the hint is satisfied, canMapVOP3PToVOPD recognises the
instruction as eligible for VOPD pairing by checking if it is VOP2 like:
dst==src2, no source modifiers, no clamp, and src1 is a register.
Mark both instructions as commutable to allow a literal in src1 to be
moved to src0, since VOPD only permits a literal in src0.

Original patch had a bug where it did not check if physical src
registers match register class of appropriate operand in fullVOPD
instructions, check is now done via isValidVOPDSrc.
---
 llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp   |  33 +-
 llvm/lib/Target/AMDGPU/SIISelLowering.cpp |   8 +
 .../Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp|   6 +
 llvm/lib/Target/AMDGPU/VOP3PInstructions.td   |  13 +-
 llvm/lib/Target/AMDGPU/VOPInstructions.td |   4 +-
 .../AMDGPU/llvm.amdgcn.fdot2.f32.bf16.ll  | 235 +++--
 llvm/test/CodeGen/AMDGPU/llvm.amdgcn.fdot2.ll | 972 --
 7 files changed, 669 insertions(+), 602 deletions(-)

diff --git a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp 
b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
index da57b0725065f..94a26440cc7c8 100644
--- a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
@@ -56,6 +56,36 @@ bool isValidVOPDSrc(const SIInstrInfo &TII, int VOPDOpc, 
unsigned CompIdx,
   return TII.getRegClass(TII.get(VOPDOpc), OpIdx)->contains(PhysSrcReg);
 }
 
+// Check if MI is a VOP3P instruction with operands that satisfy the 
constraints
+// for mapping it to a VOP2/VOPD opcode: no modifiers, no clamp, src1 and src2
+// are registers (src0 can be register or literal), and src2 is same as dst.
+static bool canMapVOP3PToVOPD(const MachineInstr &MI) {
+  unsigned Opc = MI.getOpcode();
+  if (Opc != AMDGPU::V_DOT2_F32_F16 && Opc != AMDGPU::V_DOT2_F32_BF16)
+return false;
+  // src0 can be register or literal
+  int16_t Src0ModsIdx = getNamedOperandIdx(Opc, 
AMDGPU::OpName::src0_modifiers);
+  if (MI.getOperand(Src0ModsIdx).getImm() != SISrcMods::OP_SEL_1)
+return false;
+  int16_t Src1ModsIdx = getNamedOperandIdx(Opc, 
AMDGPU::OpName::src1_modifiers);
+  if (MI.getOperand(Src1ModsIdx).getImm() != SISrcMods::OP_SEL_1)
+return false;
+  int16_t Src1Idx = getNamedOperandIdx(Opc, AMDGPU::OpName::src1);
+  if (!MI.getOperand(Src1Idx).isReg())
+return false;
+  int16_t Src2ModsIdx = getNamedOperandIdx(Opc, 
AMDGPU::OpName::src2_modifiers);
+  if (MI.getOperand(Src2ModsIdx).getImm() != SISrcMods::OP_SEL_1)
+return false;
+  int16_t Src2Idx = getNamedOperandIdx(Opc, AMDGPU::OpName::src2);
+  if (!MI.getOperand(Src2Idx).isReg())
+return false;
+  int16_t ClampIdx = getNamedOperandIdx(Opc, AMDGPU::OpName::clamp);
+  if (MI.getOperand(ClampIdx).getImm() != 0)
+return false;
+  int16_t VdstIdx = getNamedOperandIdx(Opc, AMDGPU::OpName::vdst);
+  return MI.getOperand(VdstIdx).getReg() == MI.getOperand(Src2Idx).getReg();
+}
+
 bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
const MachineInstr &MIX,
const MachineInstr &MIY, bool IsVOPD3,
@@ -67,7 +97,8 @@ bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
 
   if (IsVOPD3 && !ST.hasVOPD3())
 return false;
-  if (!IsVOPD3 && (TII.isVOP3(MIX) || TII.isVOP3(MIY)))
+  if (!IsVOPD3 && ((TII.isVOP3(MIX) && !canMapVOP3PToVOPD(MIX)) ||
+   (TII.isVOP3(MIY) && !canMapVOP3PToVOPD(MIY
 return false;
   if (TII.isDPP(MIX) || TII.isDPP(MIY))
 return false;
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp 
b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index a6dc58e6da263..96918c9f6dd36 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -7297,6 +7297,14 @@ 
SITargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
 MI.getOperand(0).setReg(OriginalExec);
 return BB;
   }
+  case AMDGPU::V_DOT2_F32_F16:
+  case AMDGPU::V_DOT2_F32_BF16: {
+// Hint RA to assign dst and src2 the same physical register.
+// For targets without VOP2, but with VOPD, variant of the instruction this
+// is one of the conditions to attempt converting VOP3P to VOPD.
+MRI.setSimpleHint(MI.getOperand(0).getReg(), MI.getOperand(6).getReg());
+return BB;
+  }
   default:
 if (TII->isImage(MI) || TII->isMUBUF(MI)) {
   if (!MI.mayStore())
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp 
b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
in

[llvm-branch-commits] [llvm] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from VOP3 (PR #196516)

2026-06-09 Thread Petar Avramovic via llvm-branch-commits 

petar-avramovic wrote:

ping

https://github.com/llvm/llvm-project/pull/196516
___
llvm-branch-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

[llvm-branch-commits] [llvm] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from VOP3 (PR #196516)

2026-06-09 Thread Petar Avramovic via llvm-branch-commits 

https://github.com/petar-avramovic updated 
https://github.com/llvm/llvm-project/pull/196516

>From 6584857c3e57e8ebe8c08e5615d0ad5ee52668d8 Mon Sep 17 00:00:00 2001
From: Petar Avramovic 
Date: Fri, 8 May 2026 11:07:09 +0200
Subject: [PATCH] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from
 VOP3

For V_DOT2_F32_F16 and V_DOT2_F32_BF16 add their VOPDName and mark
them with usesCustomInserter which will be used to add pre-RA register
allocation hints to preferably assign dst and src2 to the same physical
register. When the hint is satisfied, canMapVOP3PToVOPD recognises the
instruction as eligible for VOPD pairing by checking if it is VOP2 like:
dst==src2, no source modifiers, no clamp, and src1 is a register.
Mark both instructions as commutable to allow a literal in src1 to be
moved to src0, since VOPD only permits a literal in src0.

Original patch had a bug where it did not check if physical src
registers match register class of appropriate operand in fullVOPD
instructions, check is now done via isValidVOPDSrc.
---
 llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp   |  33 +-
 llvm/lib/Target/AMDGPU/SIISelLowering.cpp |   8 +
 .../Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp|   6 +
 llvm/lib/Target/AMDGPU/VOP3PInstructions.td   |  13 +-
 llvm/lib/Target/AMDGPU/VOPInstructions.td |   4 +-
 .../AMDGPU/llvm.amdgcn.fdot2.f32.bf16.ll  | 235 +++--
 llvm/test/CodeGen/AMDGPU/llvm.amdgcn.fdot2.ll | 972 --
 7 files changed, 669 insertions(+), 602 deletions(-)

diff --git a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp 
b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
index c5f003bd155ea..a272b49096ea2 100644
--- a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
@@ -60,6 +60,36 @@ bool isValidVOPDSrc(const SIInstrInfo &TII, int VOPDOpc, 
unsigned CompIdx,
   return TII.getRegClass(TII.get(VOPDOpc), Idx)->contains(PhysSrcReg);
 }
 
+// Check if MI is a VOP3P instruction with operands that satisfy the 
constraints
+// for mapping it to a VOP2/VOPD opcode: no modifiers, no clamp, src1 and src2
+// are registers (src0 can be register or literal), and src2 is same as dst.
+static bool canMapVOP3PToVOPD(const MachineInstr &MI) {
+  unsigned Opc = MI.getOpcode();
+  if (Opc != AMDGPU::V_DOT2_F32_F16 && Opc != AMDGPU::V_DOT2_F32_BF16)
+return false;
+  // src0 can be register or literal
+  int16_t Src0ModsIdx = getNamedOperandIdx(Opc, 
AMDGPU::OpName::src0_modifiers);
+  if (MI.getOperand(Src0ModsIdx).getImm() != SISrcMods::OP_SEL_1)
+return false;
+  int16_t Src1ModsIdx = getNamedOperandIdx(Opc, 
AMDGPU::OpName::src1_modifiers);
+  if (MI.getOperand(Src1ModsIdx).getImm() != SISrcMods::OP_SEL_1)
+return false;
+  int16_t Src1Idx = getNamedOperandIdx(Opc, AMDGPU::OpName::src1);
+  if (!MI.getOperand(Src1Idx).isReg())
+return false;
+  int16_t Src2ModsIdx = getNamedOperandIdx(Opc, 
AMDGPU::OpName::src2_modifiers);
+  if (MI.getOperand(Src2ModsIdx).getImm() != SISrcMods::OP_SEL_1)
+return false;
+  int16_t Src2Idx = getNamedOperandIdx(Opc, AMDGPU::OpName::src2);
+  if (!MI.getOperand(Src2Idx).isReg())
+return false;
+  int16_t ClampIdx = getNamedOperandIdx(Opc, AMDGPU::OpName::clamp);
+  if (MI.getOperand(ClampIdx).getImm() != 0)
+return false;
+  int16_t VdstIdx = getNamedOperandIdx(Opc, AMDGPU::OpName::vdst);
+  return MI.getOperand(VdstIdx).getReg() == MI.getOperand(Src2Idx).getReg();
+}
+
 bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
const MachineInstr &MIX,
const MachineInstr &MIY, bool IsVOPD3,
@@ -71,7 +101,8 @@ bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
 
   if (IsVOPD3 && !ST.hasVOPD3())
 return false;
-  if (!IsVOPD3 && (TII.isVOP3(MIX) || TII.isVOP3(MIY)))
+  if (!IsVOPD3 && ((TII.isVOP3(MIX) && !canMapVOP3PToVOPD(MIX)) ||
+   (TII.isVOP3(MIY) && !canMapVOP3PToVOPD(MIY
 return false;
   if (TII.isDPP(MIX) || TII.isDPP(MIY))
 return false;
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp 
b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index fb053edfc7a9e..3e2fcbde8d0c5 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -7282,6 +7282,14 @@ 
SITargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
 MI.getOperand(0).setReg(OriginalExec);
 return BB;
   }
+  case AMDGPU::V_DOT2_F32_F16:
+  case AMDGPU::V_DOT2_F32_BF16: {
+// Hint RA to assign dst and src2 the same physical register.
+// For targets without VOP2, but with VOPD, variant of the instruction this
+// is one of the conditions to attempt converting VOP3P to VOPD.
+MRI.setSimpleHint(MI.getOperand(0).getReg(), MI.getOperand(6).getReg());
+return BB;
+  }
   default:
 if (TII->isImage(MI) || TII->isMUBUF(MI)) {
   if (!MI.mayStore())
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp 
b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
ind

[llvm-branch-commits] [llvm] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from VOP3 (PR #196516)

2026-06-09 Thread Petar Avramovic via llvm-branch-commits 

https://github.com/petar-avramovic updated 
https://github.com/llvm/llvm-project/pull/196516

>From 6584857c3e57e8ebe8c08e5615d0ad5ee52668d8 Mon Sep 17 00:00:00 2001
From: Petar Avramovic 
Date: Fri, 8 May 2026 11:07:09 +0200
Subject: [PATCH] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from
 VOP3

For V_DOT2_F32_F16 and V_DOT2_F32_BF16 add their VOPDName and mark
them with usesCustomInserter which will be used to add pre-RA register
allocation hints to preferably assign dst and src2 to the same physical
register. When the hint is satisfied, canMapVOP3PToVOPD recognises the
instruction as eligible for VOPD pairing by checking if it is VOP2 like:
dst==src2, no source modifiers, no clamp, and src1 is a register.
Mark both instructions as commutable to allow a literal in src1 to be
moved to src0, since VOPD only permits a literal in src0.

Original patch had a bug where it did not check if physical src
registers match register class of appropriate operand in fullVOPD
instructions, check is now done via isValidVOPDSrc.
---
 llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp   |  33 +-
 llvm/lib/Target/AMDGPU/SIISelLowering.cpp |   8 +
 .../Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp|   6 +
 llvm/lib/Target/AMDGPU/VOP3PInstructions.td   |  13 +-
 llvm/lib/Target/AMDGPU/VOPInstructions.td |   4 +-
 .../AMDGPU/llvm.amdgcn.fdot2.f32.bf16.ll  | 235 +++--
 llvm/test/CodeGen/AMDGPU/llvm.amdgcn.fdot2.ll | 972 --
 7 files changed, 669 insertions(+), 602 deletions(-)

diff --git a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp 
b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
index c5f003bd155ea..a272b49096ea2 100644
--- a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
@@ -60,6 +60,36 @@ bool isValidVOPDSrc(const SIInstrInfo &TII, int VOPDOpc, 
unsigned CompIdx,
   return TII.getRegClass(TII.get(VOPDOpc), Idx)->contains(PhysSrcReg);
 }
 
+// Check if MI is a VOP3P instruction with operands that satisfy the 
constraints
+// for mapping it to a VOP2/VOPD opcode: no modifiers, no clamp, src1 and src2
+// are registers (src0 can be register or literal), and src2 is same as dst.
+static bool canMapVOP3PToVOPD(const MachineInstr &MI) {
+  unsigned Opc = MI.getOpcode();
+  if (Opc != AMDGPU::V_DOT2_F32_F16 && Opc != AMDGPU::V_DOT2_F32_BF16)
+return false;
+  // src0 can be register or literal
+  int16_t Src0ModsIdx = getNamedOperandIdx(Opc, 
AMDGPU::OpName::src0_modifiers);
+  if (MI.getOperand(Src0ModsIdx).getImm() != SISrcMods::OP_SEL_1)
+return false;
+  int16_t Src1ModsIdx = getNamedOperandIdx(Opc, 
AMDGPU::OpName::src1_modifiers);
+  if (MI.getOperand(Src1ModsIdx).getImm() != SISrcMods::OP_SEL_1)
+return false;
+  int16_t Src1Idx = getNamedOperandIdx(Opc, AMDGPU::OpName::src1);
+  if (!MI.getOperand(Src1Idx).isReg())
+return false;
+  int16_t Src2ModsIdx = getNamedOperandIdx(Opc, 
AMDGPU::OpName::src2_modifiers);
+  if (MI.getOperand(Src2ModsIdx).getImm() != SISrcMods::OP_SEL_1)
+return false;
+  int16_t Src2Idx = getNamedOperandIdx(Opc, AMDGPU::OpName::src2);
+  if (!MI.getOperand(Src2Idx).isReg())
+return false;
+  int16_t ClampIdx = getNamedOperandIdx(Opc, AMDGPU::OpName::clamp);
+  if (MI.getOperand(ClampIdx).getImm() != 0)
+return false;
+  int16_t VdstIdx = getNamedOperandIdx(Opc, AMDGPU::OpName::vdst);
+  return MI.getOperand(VdstIdx).getReg() == MI.getOperand(Src2Idx).getReg();
+}
+
 bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
const MachineInstr &MIX,
const MachineInstr &MIY, bool IsVOPD3,
@@ -71,7 +101,8 @@ bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
 
   if (IsVOPD3 && !ST.hasVOPD3())
 return false;
-  if (!IsVOPD3 && (TII.isVOP3(MIX) || TII.isVOP3(MIY)))
+  if (!IsVOPD3 && ((TII.isVOP3(MIX) && !canMapVOP3PToVOPD(MIX)) ||
+   (TII.isVOP3(MIY) && !canMapVOP3PToVOPD(MIY
 return false;
   if (TII.isDPP(MIX) || TII.isDPP(MIY))
 return false;
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp 
b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index fb053edfc7a9e..3e2fcbde8d0c5 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -7282,6 +7282,14 @@ 
SITargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
 MI.getOperand(0).setReg(OriginalExec);
 return BB;
   }
+  case AMDGPU::V_DOT2_F32_F16:
+  case AMDGPU::V_DOT2_F32_BF16: {
+// Hint RA to assign dst and src2 the same physical register.
+// For targets without VOP2, but with VOPD, variant of the instruction this
+// is one of the conditions to attempt converting VOP3P to VOPD.
+MRI.setSimpleHint(MI.getOperand(0).getReg(), MI.getOperand(6).getReg());
+return BB;
+  }
   default:
 if (TII->isImage(MI) || TII->isMUBUF(MI)) {
   if (!MI.mayStore())
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp 
b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
ind

[llvm-branch-commits] [llvm] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from VOP3 (PR #196516)

2026-05-08 Thread Petar Avramovic via llvm-branch-commits 

https://github.com/petar-avramovic created 
https://github.com/llvm/llvm-project/pull/196516

For V_DOT2_F32_F16 and V_DOT2_F32_BF16 add their VOPDName and mark
them with usesCustomInserter which will be used to add pre-RA register
allocation hints to preferably assign dst and src2 to the same physical
register. When the hint is satisfied, canMapVOP3PToVOPD recognises the
instruction as eligible for VOPD pairing by checking if it is VOP2 like:
dst==src2, no source modifiers, no clamp, and src1 is a register.
Mark both instructions as commutable to allow a literal in src1 to be
moved to src0, since VOPD only permits a literal in src0.

Original patch had a bug where it did not check if physical src
registers match register class of appropriate operand in fullVOPD
instructions, check is now done via isValidVOPDSrc.

>From 3ba58a54cb984defd100f5afa87d92465c6befb4 Mon Sep 17 00:00:00 2001
From: Petar Avramovic 
Date: Fri, 8 May 2026 11:07:09 +0200
Subject: [PATCH] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from
 VOP3

For V_DOT2_F32_F16 and V_DOT2_F32_BF16 add their VOPDName and mark
them with usesCustomInserter which will be used to add pre-RA register
allocation hints to preferably assign dst and src2 to the same physical
register. When the hint is satisfied, canMapVOP3PToVOPD recognises the
instruction as eligible for VOPD pairing by checking if it is VOP2 like:
dst==src2, no source modifiers, no clamp, and src1 is a register.
Mark both instructions as commutable to allow a literal in src1 to be
moved to src0, since VOPD only permits a literal in src0.

Original patch had a bug where it did not check if physical src
registers match register class of appropriate operand in fullVOPD
instructions, check is now done via isValidVOPDSrc.
---
 llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp   |  33 +-
 llvm/lib/Target/AMDGPU/SIISelLowering.cpp |   8 +
 .../Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp|   6 +
 llvm/lib/Target/AMDGPU/VOP3PInstructions.td   |  13 +-
 llvm/lib/Target/AMDGPU/VOPInstructions.td |   4 +-
 .../AMDGPU/llvm.amdgcn.fdot2.f32.bf16.ll  | 235 +++--
 llvm/test/CodeGen/AMDGPU/llvm.amdgcn.fdot2.ll | 972 --
 7 files changed, 669 insertions(+), 602 deletions(-)

diff --git a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp 
b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
index fb26175769d94..254569d20113b 100644
--- a/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNVOPDUtils.cpp
@@ -60,6 +60,36 @@ bool isValidVOPDSrc(const SIInstrInfo &TII, int VOPDOpc, 
unsigned CompIdx,
   return TII.getRegClass(TII.get(VOPDOpc), Idx)->contains(PhysSrcReg);
 }
 
+// Check if MI is a VOP3P instruction with operands that satisfy the 
constraints
+// for mapping it to a VOP2/VOPD opcode: no modifiers, no clamp, src1 and src2
+// are registers (src0 can be register or literal), and src2 is same as dst.
+static bool canMapVOP3PToVOPD(const MachineInstr &MI) {
+  unsigned Opc = MI.getOpcode();
+  if (Opc != AMDGPU::V_DOT2_F32_F16 && Opc != AMDGPU::V_DOT2_F32_BF16)
+return false;
+  // src0 can be register or literal
+  int16_t Src0ModsIdx = getNamedOperandIdx(Opc, 
AMDGPU::OpName::src0_modifiers);
+  if (MI.getOperand(Src0ModsIdx).getImm() != SISrcMods::OP_SEL_1)
+return false;
+  int16_t Src1ModsIdx = getNamedOperandIdx(Opc, 
AMDGPU::OpName::src1_modifiers);
+  if (MI.getOperand(Src1ModsIdx).getImm() != SISrcMods::OP_SEL_1)
+return false;
+  int16_t Src1Idx = getNamedOperandIdx(Opc, AMDGPU::OpName::src1);
+  if (!MI.getOperand(Src1Idx).isReg())
+return false;
+  int16_t Src2ModsIdx = getNamedOperandIdx(Opc, 
AMDGPU::OpName::src2_modifiers);
+  if (MI.getOperand(Src2ModsIdx).getImm() != SISrcMods::OP_SEL_1)
+return false;
+  int16_t Src2Idx = getNamedOperandIdx(Opc, AMDGPU::OpName::src2);
+  if (!MI.getOperand(Src2Idx).isReg())
+return false;
+  int16_t ClampIdx = getNamedOperandIdx(Opc, AMDGPU::OpName::clamp);
+  if (MI.getOperand(ClampIdx).getImm() != 0)
+return false;
+  int16_t VdstIdx = getNamedOperandIdx(Opc, AMDGPU::OpName::vdst);
+  return MI.getOperand(VdstIdx).getReg() == MI.getOperand(Src2Idx).getReg();
+}
+
 bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
const MachineInstr &MIX,
const MachineInstr &MIY, bool IsVOPD3,
@@ -71,7 +101,8 @@ bool llvm::checkVOPDRegConstraints(const SIInstrInfo &TII,
 
   if (IsVOPD3 && !ST.hasVOPD3())
 return false;
-  if (!IsVOPD3 && (TII.isVOP3(MIX) || TII.isVOP3(MIY)))
+  if (!IsVOPD3 && ((TII.isVOP3(MIX) && !canMapVOP3PToVOPD(MIX)) ||
+   (TII.isVOP3(MIY) && !canMapVOP3PToVOPD(MIY
 return false;
   if (TII.isDPP(MIX) || TII.isDPP(MIY))
 return false;
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp 
b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index 20599228beea8..92642165c161b 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -7223

[llvm-branch-commits] [llvm] AMDGPU: Reland: Codegen for v_dual_dot2acc_f32_f16/bf16 from VOP3 (PR #196516)

2026-05-08 Thread Petar Avramovic via llvm-branch-commits 

petar-avramovic wrote:

> [!WARNING]
> This pull request is not mergeable via GitHub because a downstack PR is 
> open. Once all requirements are satisfied, merge this PR as a stack  href="https://app.graphite.com/github/pr/llvm/llvm-project/196516?utm_source=stack-comment-downstack-mergeability-warning";
>  >on Graphite.
> https://graphite.dev/docs/merge-pull-requests";>Learn more

* **#196516** https://app.graphite.com/github/pr/llvm/llvm-project/196516?utm_source=stack-comment-icon";
 target="_blank">https://static.graphite.dev/graphite-32x32-black.png"; alt="Graphite" 
width="10px" height="10px"/> 👈 https://app.graphite.com/github/pr/llvm/llvm-project/196516?utm_source=stack-comment-view-in-graphite";
 target="_blank">(View in Graphite)
* **#196515** https://app.graphite.com/github/pr/llvm/llvm-project/196515?utm_source=stack-comment-icon";
 target="_blank">https://static.graphite.dev/graphite-32x32-black.png"; alt="Graphite" 
width="10px" height="10px"/>
* **#196514** https://app.graphite.com/github/pr/llvm/llvm-project/196514?utm_source=stack-comment-icon";
 target="_blank">https://static.graphite.dev/graphite-32x32-black.png"; alt="Graphite" 
width="10px" height="10px"/>
* `main`




This stack of pull requests is managed by https://graphite.dev?utm-source=stack-comment";>Graphite. Learn 
more about https://stacking.dev/?utm_source=stack-comment";>stacking.


https://github.com/llvm/llvm-project/pull/196516
___
llvm-branch-commits mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits