llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT--> @llvm/pr-subscribers-llvm-ir Author: Ganesh (ganeshgit) <details> <summary>Changes</summary> ACE v1 introduces new tile-based matrix operations with fixed 16x64 tile dimensions (Palette 2), unlike AMX's configurable dimensions. Key features: - New __acetile type with fixed 16x64 dimensions - __tile_ace_* intrinsics for ACE tile operations - Outer product instructions: TOP4BUUD, TOP4BUSD, TOP4BSSD, TOP4BSUD, TOP2BF16PS - Mixed precision FP8 instructions: TOP4MX variants - Tile movement: TILEMOVROW, TILEMOVCOL for ZMM<->tile transfers - BSR (Block Scale Register) operations for scaling factors - ACE tile spill/reload using TILEMOVROW + VMOVUPS row-by-row (ACE doesn't have TILELOADD/TILESTORED instructions) Backend changes: - X86LowerTileCopy handles ACE tile copies - X86PreTileConfig/X86FastPreTileConfig for ACE register allocation - Palette 2 tile configuration support Reused from AMX: - The x86_amx type - AMX-AVX512f definitions and TILE instruction definitions - Passes for lowering and handling the x86_amx type - Wrappers internally use rows and cols to use amx definitions This PR builds on PR #<!-- -->206888 and should be reviewed after/with it. --- Patch is 228.20 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/208408.diff 58 Files Affected: - (modified) clang/include/clang/Basic/BuiltinsX86_64.td (+68-3) - (modified) clang/include/clang/Options/Options.td (+2) - (modified) clang/lib/Basic/Targets/X86.cpp (+6) - (modified) clang/lib/Basic/Targets/X86.h (+1) - (modified) clang/lib/Headers/CMakeLists.txt (+1) - (added) clang/lib/Headers/acev1intrin.h (+766) - (modified) clang/lib/Headers/cpuid.h (+2) - (modified) clang/lib/Headers/immintrin.h (+2) - (modified) clang/lib/Sema/SemaX86.cpp (+5) - (added) clang/test/CodeGen/X86/ace-api.c (+145) - (added) clang/test/CodeGen/X86/ace-builtins.c (+119) - (added) clang/test/CodeGen/X86/ace-inline-asm.c (+85) - (added) clang/test/CodeGen/X86/acev1.c (+38) - (added) clang/test/CodeGen/X86/acev1_api.c (+138) - (added) clang/test/CodeGen/X86/acev1_bsr.c (+63) - (added) clang/test/CodeGen/X86/acev1_errors.c (+17) - (added) clang/test/CodeGen/X86/acev1_inline_asm.c (+76) - (added) clang/test/CodeGen/X86/acev1_mxfp8.c (+51) - (added) clang/test/CodeGen/X86/acev1_tile_movement.c (+113) - (modified) clang/test/CodeGen/attr-target-x86.c (+2-2) - (modified) clang/test/Preprocessor/x86_target_features.c (+7) - (modified) llvm/include/llvm/IR/IntrinsicsX86.td (+198) - (modified) llvm/include/llvm/TargetParser/X86TargetParser.def (+1) - (modified) llvm/lib/Target/X86/AsmParser/X86AsmParser.cpp (+50) - (modified) llvm/lib/Target/X86/MCTargetDesc/X86ATTInstPrinter.cpp (+90-2) - (modified) llvm/lib/Target/X86/MCTargetDesc/X86ATTInstPrinter.h (+1) - (modified) llvm/lib/Target/X86/MCTargetDesc/X86IntelInstPrinter.cpp (+87-1) - (modified) llvm/lib/Target/X86/MCTargetDesc/X86IntelInstPrinter.h (+1) - (modified) llvm/lib/Target/X86/X86.td (+3) - (modified) llvm/lib/Target/X86/X86ExpandPseudo.cpp (+89) - (modified) llvm/lib/Target/X86/X86FastPreTileConfig.cpp (+80-25) - (modified) llvm/lib/Target/X86/X86FastTileConfig.cpp (+13-4) - (modified) llvm/lib/Target/X86/X86FrameLowering.cpp (+3-2) - (modified) llvm/lib/Target/X86/X86ISelDAGToDAG.cpp (+19) - (modified) llvm/lib/Target/X86/X86ISelLowering.cpp (+282-8) - (added) llvm/lib/Target/X86/X86InstrACE.td (+417) - (modified) llvm/lib/Target/X86/X86InstrAMX.td (+44-23) - (modified) llvm/lib/Target/X86/X86InstrInfo.cpp (+7-1) - (modified) llvm/lib/Target/X86/X86InstrInfo.td (+3) - (modified) llvm/lib/Target/X86/X86InstrPredicates.td (+3) - (modified) llvm/lib/Target/X86/X86LowerAMXType.cpp (+143-2) - (modified) llvm/lib/Target/X86/X86LowerTileCopy.cpp (+152-48) - (modified) llvm/lib/Target/X86/X86MachineFunctionInfo.cpp (+3-1) - (modified) llvm/lib/Target/X86/X86MachineFunctionInfo.h (+22) - (modified) llvm/lib/Target/X86/X86PreTileConfig.cpp (+7-3) - (modified) llvm/lib/Target/X86/X86RegisterInfo.cpp (+15-2) - (modified) llvm/lib/Target/X86/X86RegisterInfo.td (+9) - (modified) llvm/lib/Target/X86/X86TileConfig.cpp (+8-2) - (modified) llvm/lib/TargetParser/X86TargetParser.cpp (+1) - (added) llvm/test/CodeGen/X86/ACE/ace-intrinsics.ll (+244) - (added) llvm/test/CodeGen/X86/ACE/acev1-greedy-ra.ll (+99) - (added) llvm/test/CodeGen/X86/ACE/acev1-outer-product.ll (+204) - (added) llvm/test/CodeGen/X86/ACE/acev1-tile-basic.ll (+94) - (modified) llvm/test/CodeGen/X86/ipra-reg-usage.ll (+2-2) - (added) llvm/test/MC/X86/ACE/ace-att.s (+109) - (added) llvm/test/MC/X86/ACE/ace-error.s (+39) - (added) llvm/test/MC/X86/ACE/ace-intel.s (+109) - (modified) llvm/test/TableGen/x86-fold-tables.inc (+5) ``````````diff diff --git a/clang/include/clang/Basic/BuiltinsX86_64.td b/clang/include/clang/Basic/BuiltinsX86_64.td index 0a98b3049df34..6f34256139144 100644 --- a/clang/include/clang/Basic/BuiltinsX86_64.td +++ b/clang/include/clang/Basic/BuiltinsX86_64.td @@ -223,6 +223,9 @@ let Features = "amx-int8", Attributes = [NoThrow] in { let Features = "amx-tile", Attributes = [NoThrow] in { def tilestored64_internal : X86Builtin<"void(unsigned short, unsigned short, void *, size_t, _Vector<256, int>)">; +} + +let Features = "amx-tile|acev1", Attributes = [NoThrow] in { def tilezero_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short)">; } @@ -239,7 +242,7 @@ let Features = "amx-complex", Attributes = [NoThrow] in { def tcmmrlfp16ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Vector<256, int>, _Vector<256, int>, _Vector<256, int>)">; } -let Features = "amx-avx512,avx10.2", Attributes = [NoThrow, RequiredVectorWidth<512>] in { +let Features = "amx-avx512,avx10.2|acev1", Attributes = [NoThrow, RequiredVectorWidth<512>] in { def tcvtrowd2ps_internal : X86Builtin<"_Vector<16, float>(unsigned short, unsigned short, _Vector<256, int>, unsigned int)">; def tcvtrowps2bf16h_internal : X86Builtin<"_Vector<32, __bf16>(unsigned short, unsigned short, _Vector<256, int>, unsigned int)">; def tcvtrowps2bf16l_internal : X86Builtin<"_Vector<32, __bf16>(unsigned short, unsigned short, _Vector<256, int>, unsigned int)">; @@ -255,7 +258,7 @@ let Features = "amx-fp8", Attributes = [NoThrow] in { def tdphf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Vector<256, int>, _Vector<256, int>, _Vector<256, int>)">; } -let Features = "amx-tile", Attributes = [NoThrow] in { +let Features = "amx-tile|acev1", Attributes = [NoThrow] in { def tile_loadconfig : X86Builtin<"void(void const *)">; def tile_storeconfig : X86Builtin<"void(void const *)">; def tilerelease : X86Builtin<"void()">; @@ -293,7 +296,7 @@ let Features = "amx-complex", Attributes = [NoThrow] in { def tcmmrlfp16ps : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">; } -let Features = "amx-avx512,avx10.2", Attributes = [NoThrow, RequiredVectorWidth<512>] in { +let Features = "amx-avx512,avx10.2|acev1", Attributes = [NoThrow, RequiredVectorWidth<512>] in { def tcvtrowd2ps : X86Builtin<"_Vector<16, float>(_Constant unsigned char, unsigned int)">; def tcvtrowps2bf16h : X86Builtin<"_Vector<32, __bf16>(_Constant unsigned char, unsigned int)">; def tcvtrowps2bf16l : X86Builtin<"_Vector<32, __bf16>(_Constant unsigned char, unsigned int)">; @@ -389,3 +392,65 @@ let Features = "movrs,avx10.2", Attributes = [NoThrow, RequiredVectorWidth<256>] let Features = "movrs,avx10.2", Attributes = [NoThrow, RequiredVectorWidth<512>] in { def vmovrsw512 : X86Builtin<"_Vector<32, short>(_Vector<32, short const *>)">; } + +// ACE (AI Compute Extensions) Builtins +// Tile movement - single builtin handles both immediate and register index forms +// The lowering code checks if index is constant and selects appropriate instruction +let Features = "acev1", Attributes = [NoThrow] in { + def tilesetcol : X86Builtin<"void(_Constant unsigned char, _Vector<16, int>, unsigned int)">; + // ACE tilemovrow writes TO tile (AMX tilemovrow reads FROM tile) + def tilesetrow : X86Builtin<"void(_Constant unsigned char, _Vector<16, int>, unsigned int)">; +} + +// Tile movement internal (for IR-level operations) +let Features = "acev1", Attributes = [NoThrow] in { + def tilesetcol_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, _Vector<16, int>, unsigned int)">; + // ACE tilemovrow_to_tile writes TO tile (AMX tilemovrow reads FROM tile) + def tilesetrow_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, _Vector<16, int>, unsigned int)">; +} + +// BSR management +let Features = "acev1", Attributes = [NoThrow] in { + def bsrinit : X86Builtin<"void()">; + def bsrmovf : X86Builtin<"void(_Vector<16, int>, _Vector<16, int>)">; + def bsrmovh_set : X86Builtin<"void(_Vector<16, int>)">; + def bsrmovh_get : X86Builtin<"_Vector<16, int>()">; + def bsrmovl_set : X86Builtin<"void(_Vector<16, int>)">; + def bsrmovl_get : X86Builtin<"_Vector<16, int>()">; +} + +// Outer products - user-facing (immediate tile IDs) +let Features = "acev1", Attributes = [NoThrow] in { + def top2bf16ps : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">; + def top4buud : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">; + def top4busd : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">; + def top4bssd : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">; + def top4bsud : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">; +} + +// Mixed precision outer products - user-facing +let Features = "acev1", Attributes = [NoThrow] in { + def top4mxhf8ps : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">; + def top4mxbhf8ps : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">; + def top4mxhbf8ps : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">; + def top4mxbf8ps : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">; + def top4mxbssps : X86Builtin<"void(_Constant unsigned char, _Constant unsigned char, _Constant unsigned char, _Constant unsigned char)">; +} + +// Outer products - internal (for IR-level operations with tile values) +let Features = "acev1", Attributes = [NoThrow] in { + def top2bf16ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Vector<256, int>, _Vector<32, __bf16>, _Vector<32, __bf16>)">; + def top4buud_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Vector<256, int>, _Vector<64, signed char>, _Vector<64, signed char>)">; + def top4busd_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Vector<256, int>, _Vector<64, signed char>, _Vector<64, signed char>)">; + def top4bssd_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Vector<256, int>, _Vector<64, signed char>, _Vector<64, signed char>)">; + def top4bsud_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, _Vector<256, int>, _Vector<64, signed char>, _Vector<64, signed char>)">; +} + +// Mixed precision internal +let Features = "acev1", Attributes = [NoThrow] in { + def top4mxhf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">; + def top4mxbhf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">; + def top4mxhbf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">; + def top4mxbf8ps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">; + def top4mxbssps_internal : X86Builtin<"_Vector<256, int>(unsigned short, unsigned short, unsigned short, unsigned char, _Vector<256, int>, _Vector<16, int>, _Vector<16, int>)">; +} diff --git a/clang/include/clang/Options/Options.td b/clang/include/clang/Options/Options.td index 59bfd621a5994..c6ab7b6468802 100644 --- a/clang/include/clang/Options/Options.td +++ b/clang/include/clang/Options/Options.td @@ -7178,6 +7178,8 @@ def mamx_tile : Flag<["-"], "mamx-tile">, Group<m_x86_Features_Group>; def mno_amx_tile : Flag<["-"], "mno-amx-tile">, Group<m_x86_Features_Group>; def mamx_movrs: Flag<["-"], "mamx-movrs">, Group<m_x86_Features_Group>; def mno_amx_movrs: Flag<["-"], "mno-amx-movrs">, Group<m_x86_Features_Group>; +def macev1 : Flag<["-"], "macev1">, Group<m_x86_Features_Group>; +def mno_acev1 : Flag<["-"], "mno-acev1">, Group<m_x86_Features_Group>; def mcmpccxadd : Flag<["-"], "mcmpccxadd">, Group<m_x86_Features_Group>; def mno_cmpccxadd : Flag<["-"], "mno-cmpccxadd">, Group<m_x86_Features_Group>; def msse : Flag<["-"], "msse">, Group<m_x86_Features_Group>; diff --git a/clang/lib/Basic/Targets/X86.cpp b/clang/lib/Basic/Targets/X86.cpp index f77b2d4e0815d..c694458b1d1ff 100644 --- a/clang/lib/Basic/Targets/X86.cpp +++ b/clang/lib/Basic/Targets/X86.cpp @@ -400,6 +400,8 @@ bool X86TargetInfo::handleTargetFeatures(std::vector<std::string> &Features, HasAMXMOVRS = true; } else if (Feature == "+amx-avx512") { HasAMXAVX512 = true; + } else if (Feature == "+acev1") { + HasACEV1 = true; } else if (Feature == "+cmpccxadd") { HasCMPCCXADD = true; } else if (Feature == "+raoint") { @@ -948,6 +950,8 @@ void X86TargetInfo::getTargetDefines(const LangOptions &Opts, Builder.defineMacro("__AMX_MOVRS__"); if (HasAMXAVX512) Builder.defineMacro("__AMX_AVX512__"); + if (HasACEV1) + Builder.defineMacro("__ACEV1__"); if (HasCMPCCXADD) Builder.defineMacro("__CMPCCXADD__"); if (HasRAOINT) @@ -1087,6 +1091,7 @@ bool X86TargetInfo::isValidFeatureName(StringRef Name) const { .Case("amx-int8", true) .Case("amx-movrs", true) .Case("amx-tile", true) + .Case("acev1", true) .Case("avx", true) .Case("avx10.1", true) .Case("avx10.2", true) @@ -1208,6 +1213,7 @@ bool X86TargetInfo::hasFeature(StringRef Feature) const { .Case("amx-int8", HasAMXINT8) .Case("amx-movrs", HasAMXMOVRS) .Case("amx-tile", HasAMXTILE) + .Case("acev1", HasACEV1) .Case("avx", SSELevel >= AVX) .Case("avx10.1", HasAVX10_1) .Case("avx10.2", HasAVX10_2) diff --git a/clang/lib/Basic/Targets/X86.h b/clang/lib/Basic/Targets/X86.h index e305d9017d897..c787f8e77b185 100644 --- a/clang/lib/Basic/Targets/X86.h +++ b/clang/lib/Basic/Targets/X86.h @@ -164,6 +164,7 @@ class LLVM_LIBRARY_VISIBILITY X86TargetInfo : public TargetInfo { bool HasAMXFP8 = false; bool HasAMXMOVRS = false; bool HasAMXAVX512 = false; + bool HasACEV1 = false; bool HasSERIALIZE = false; bool HasTSXLDTRK = false; bool HasUSERMSR = false; diff --git a/clang/lib/Headers/CMakeLists.txt b/clang/lib/Headers/CMakeLists.txt index 4bbfb9ba1662a..a64952e4d87e3 100644 --- a/clang/lib/Headers/CMakeLists.txt +++ b/clang/lib/Headers/CMakeLists.txt @@ -167,6 +167,7 @@ set(webassembly_files set(x86_files # Intrinsics + acev1intrin.h adcintrin.h adxintrin.h ammintrin.h diff --git a/clang/lib/Headers/acev1intrin.h b/clang/lib/Headers/acev1intrin.h new file mode 100644 index 0000000000000..0491c7f0b383d --- /dev/null +++ b/clang/lib/Headers/acev1intrin.h @@ -0,0 +1,766 @@ +/*===-------------- acev1intrin.h - ACEV1 intrinsics -*- C/C++ -*------------=== + * + * Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. + * See https://llvm.org/LICENSE.txt for license information. + * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + * + *===------------------------------------------------------------------------=== + */ + +#ifndef __IMMINTRIN_H +#error "Never use <acev1intrin.h> directly; include <immintrin.h> instead." +#endif /* __IMMINTRIN_H */ + +#ifndef __ACEV1INTRIN_H +#define __ACEV1INTRIN_H +#ifdef __x86_64__ + +/* Define the default attributes for the functions in this file. */ +#define __DEFAULT_FN_ATTRS_ACE \ + __attribute__((__always_inline__, __nodebug__, __target__("acev1"))) + +/// Load tile configuration from a 64-byte memory location. For ACE +/// (Palette 2), the palette_id byte must be 2. Unlike AMX (Palette 1), +/// ACE tiles have fixed dimensions of 16 rows × 64 bytes, so per-tile +/// row/column configuration bytes are ignored. If palette_id is zero, +/// tiles return to init state and are zeroed. Invalid configurations +/// result in #GP fault. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsic corresponds to the <c> LDTILECFG </c> instruction. +/// +/// \param __config +/// A pointer to 64-byte tile configuration (use __ace_tile_config). +static __inline__ void __DEFAULT_FN_ATTRS_ACE +_tile_ace_loadconfig(const void *__config) { + __builtin_ia32_tile_loadconfig(__config); +} + +/// Store the current tile configuration to a 64-byte memory location. +/// For ACE (Palette 2), the stored palette_id will be 2 and per-tile +/// configuration bytes reflect the fixed 16×64 dimensions. If tiles +/// are not configured, all zeroes are stored. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsic corresponds to the <c> STTILECFG </c> instruction. +/// +/// \param __config +/// A pointer to 64-byte tile configuration buffer. +static __inline__ void __DEFAULT_FN_ATTRS_ACE +_tile_ace_storeconfig(void *__config) { + __builtin_ia32_tile_storeconfig(__config); +} + +/// Release the tile configuration to return to init state, releasing +/// all tile storage. After this, tiles must be reconfigured with +/// _tile_ace_loadconfig before use. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsic corresponds to the <c> TILERELEASE </c> instruction. +static __inline__ void __DEFAULT_FN_ATTRS_ACE _tile_ace_release(void) { + __builtin_ia32_tilerelease(); +} + +/// Zero the ACE tile specified by "tile". Sets all 1024 bytes +/// (16 rows × 64 bytes) of the tile register to zero. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsic corresponds to the <c> TILEZERO </c> instruction. +/// +/// \param tile +/// Destination tile register ID (0-7). +#define _tile_ace_zero(tile) __builtin_ia32_tilezero((tile)) + +/// Move a 64-byte ZMM vector to a tile column. The 16 doublewords from +/// the ZMM are written as a vertical column in the tile at the specified +/// index. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsic corresponds to the <c> TILEMOVCOL </c> instruction. +/// +/// \param dst +/// Destination tile register ID (0-7). +/// \param src +/// Source ZMM register ID (0-31). +/// \param idx +/// Column index (0-15). Immediate or register form selected automatically. +#define _tile_setcol(dst, src, idx) \ + __builtin_ia32_tilesetcol((dst), (src), (idx)) + +/// Move a 64-byte ZMM vector to a tile row. The ZMM contents are written +/// as a horizontal row in the tile at the specified index. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsic corresponds to the <c> TILEMOVROW </c> instruction. +/// +/// \param dst +/// Destination tile register ID (0-7). +/// \param src +/// Source ZMM register ID (0-31). +/// \param idx +/// Row index (0-15). Immediate or register form selected automatically. +#define _tile_setrow(dst, src, idx) \ + __builtin_ia32_tilesetrow((dst), (src), (idx)) + +/// Initialize the Block Scale Register (BSR) to zero. The BSR holds +/// 32 scaling factors used by mixed-precision outer product instructions +/// (TOP4MX* variants). Must be called before using BSR-scaled operations. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsic corresponds to the <c> BSRINIT </c> instruction. +static __inline__ void __DEFAULT_FN_ATTRS_ACE _bsr0_init(void) { + __builtin_ia32_bsrinit(); +} + +/// Load the full BSR (32 scale factors) from two ZMM registers. The low +/// half (16 factors) comes from __src1, high half from __src2. Each +/// doubleword contains one scale factor. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsic corresponds to the <c> BSRMOVF </c> instruction. +/// +/// \param __src1 +/// ZMM with scale factors for BSR low half (factors 0-15). +/// \param __src2 +/// ZMM with scale factors for BSR high half (factors 16-31). +static __inline__ void __DEFAULT_FN_ATTRS_ACE _bsr0_movf(__m512i __src1, + __m512i __src2) { + __builtin_ia32_bsrmovf((__v16si)__src1, (__v16si)__src2); +} + +/// Load the high half of BSR (scale factors 16-31) from a ZMM register. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsic corresponds to the <c> BSRMOVH </c> instruction. +/// +/// \param __src +/// ZMM with 16 scale factors to write to BSR high half. +static __inline__ void __DEFAULT_FN_ATTRS_ACE _bsr0_movh_set(__m512i __src) { + __builtin_ia32_bsrmovh_set((__v16si)__src); +} + +/// Read the high half of BSR (scale factors 16-31) to a ZMM register. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsic corresponds to the <c> BSRMOVH </c> instruction. +/// +/// \returns +/// ZMM containing 16 scale factors from BSR high half. +static __inline__ __m512i __DEFAULT_FN_ATTRS_ACE _bsr0_movh_get(void) { + return (__m512i)__builtin_ia32_bsrmovh_get(); +} + +/// Load the low half of BSR (scale factors 0-15) from a ZMM register. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsic corresponds to the <c> BSRMOVL </c> instruction. +/// +/// \param __src +/// ZMM with 16 scale factors to write to BSR low half. +static __inline__ void __DEFAULT_FN_ATTRS_ACE _bsr0_movl_set(__m512i __src) { + __builtin_ia32_bsrmovl_set((__v16si)__src); +} + +/// Read the low half of BSR (scale factors 0-15) to a ZMM register. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsic corresponds to the <c> BSRMOVL </c> instruction. +/// +/// \returns +/// ZMM containing 16 scale factors from BSR low half. +static __inline__ __m512i __DEFAULT_FN_ATTRS_ACE _bsr0_movl_get(void) { + return (__m512i)__builtin_ia32_bsrmovl_get(); +} + +/// Compute 2-way outer product of BF16 pairs, accumulating to FP32. +/// Each BF16 pair from src1 and src2 produces two FP32 products that +/// are accumulated into the destination tile. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsic corresponds to the <c> TOP2BF16PS </c> instruction. +/// +/// \param dst +/// Destination/accumulator tile register ID (0-7). +/// \param src1 +/// First source ZMM register ID (0-31) containing 32 BF16 values. +/// \param src2 +/// Second source ZMM register ID (0-31) containing 32 BF16 values. +#define _tile_top2bf16ps(dst, src1, src2) \ + __builtin_ia32_top2bf16ps((dst), (src1), (src2)) + +/// Compute 4-way outer product of unsigned×unsigned bytes to INT32. +/// Each group of 4 unsigned byte pairs from src1 and src2 produces +/// 4 products accumulated into the destination tile as 32-bit integers. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsic corresponds to the <c> TOP4BUUD </c> instruction. +/// +/// \param dst +/// Destination/accumulator tile register ID (0-7). +/// \param src1 +/// First source ZMM register ID (0-31) containing 64 unsigned bytes. +/// \param src2 +/// Second source ZMM register ID (0-31) containing 64 unsigned bytes. +#define _tile_top4buud(dst, src1, src2) \ + __builtin_ia32_top4buud((dst), (src1), (src2)) + +/// Compute 4-way outer product of unsigned×signed bytes to INT32. +/// Each group of 4 byte pairs (unsigned from src1, signed from src2) +/// produces 4 products accumulated into the destination tile. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsic corresponds to the <c> TOP4BUSD </c> instruction. +/// +/// \param dst +/// Destination/accumulator tile register ID (0-7). +/// \param src1 +/// First source ZMM register ID (0-31) containing 64 unsigned bytes. +/// \param src2 +/// Second source ZMM register ID (0-31) containing 64 signed bytes. +#define _tile_top4busd(dst, src1, src2) \ + __builtin_ia32_top4busd((dst), (src1), (src2)) + +/// Compute 4-way outer product of signed×signed bytes to INT32. +/// Each group of 4 signed byte pairs produces 4 products accumulated +/// into the destination tile as 32-bit integers. +/// +/// \headerfile <immintrin.h> +/// +/// This intrinsi... [truncated] `````````` </details> https://github.com/llvm/llvm-project/pull/208408 _______________________________________________ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
