Hi Stam, > -----Original Message----- > From: Stam Markianos-Wright <stam.markianos-wri...@arm.com> > Sent: Wednesday, September 6, 2023 6:19 PM > To: gcc-patches@gcc.gnu.org > Cc: Kyrylo Tkachov <kyrylo.tkac...@arm.com>; Richard Earnshaw > <richard.earns...@arm.com> > Subject: [PING][PATCH 1/2] arm: Add define_attr to to create a mapping > between MVE predicated and unpredicated insns > > > Hi all, > > I'd like to submit two patches that add support for Arm's MVE > Tail Predicated Low Overhead Loop feature. > > --- Introduction --- > > The M-class Arm-ARM: > https://developer.arm.com/documentation/ddi0553/bu/?lang=en > Section B5.5.1 "Loop tail predication" describes the feature > we are adding support for with this patch (although > we only add codegen for DLSTP/LETP instruction loops). > > Previously with commit d2ed233cb94 we'd added support for > non-MVE DLS/LE loops through the loop-doloop pass, which, given > a standard MVE loop like: > > ``` > void __attribute__ ((noinline)) test (int16_t *a, int16_t *b, int16_t > *c, int n) > { > while (n > 0) > { > mve_pred16_t p = vctp16q (n); > int16x8_t va = vldrhq_z_s16 (a, p); > int16x8_t vb = vldrhq_z_s16 (b, p); > int16x8_t vc = vaddq_x_s16 (va, vb, p); > vstrhq_p_s16 (c, vc, p); > c+=8; > a+=8; > b+=8; > n-=8; > } > } > ``` > .. would output: > > ``` > <pre-calculate the number of iterations and place it into lr> > dls lr, lr > .L3: > vctp.16 r3 > vmrs ip, P0 @ movhi > sxth ip, ip > vmsr P0, ip @ movhi > mov r4, r0 > vpst > vldrht.16 q2, [r4] > mov r4, r1 > vmov q3, q0 > vpst > vldrht.16 q1, [r4] > mov r4, r2 > vpst > vaddt.i16 q3, q2, q1 > subs r3, r3, #8 > vpst > vstrht.16 q3, [r4] > adds r0, r0, #16 > adds r1, r1, #16 > adds r2, r2, #16 > le lr, .L3 > ``` > > where the LE instruction will decrement LR by 1, compare and > branch if needed. > > (there are also other inefficiencies with the above code, like the > pointless vmrs/sxth/vmsr on the VPR and the adds not being merged > into the vldrht/vstrht as a #16 offsets and some random movs! > But that's different problems...) > > The MVE version is similar, except that: > * Instead of DLS/LE the instructions are DLSTP/LETP. > * Instead of pre-calculating the number of iterations of the > loop, we place the number of elements to be processed by the > loop into LR. > * Instead of decrementing the LR by one, LETP will decrement it > by FPSCR.LTPSIZE, which is the number of elements being > processed in each iteration: 16 for 8-bit elements, 5 for 16-bit > elements, etc. > * On the final iteration, automatic Loop Tail Predication is > performed, as if the instructions within the loop had been VPT > predicated with a VCTP generating the VPR predicate in every > loop iteration. > > The dlstp/letp loop now looks like: > > ``` > <place n into r3> > dlstp.16 lr, r3 > .L14: > mov r3, r0 > vldrh.16 q3, [r3] > mov r3, r1 > vldrh.16 q2, [r3] > mov r3, r2 > vadd.i16 q3, q3, q2 > adds r0, r0, #16 > vstrh.16 q3, [r3] > adds r1, r1, #16 > adds r2, r2, #16 > letp lr, .L14 > > ``` > > Since the loop tail predication is automatic, we have eliminated > the VCTP that had been specified by the user in the intrinsic > and converted the VPT-predicated instructions into their > unpredicated equivalents (which also saves us from VPST insns). > > The LE instruction here decrements LR by 8 in each iteration. > > --- This 1/2 patch --- > > This first patch lays some groundwork by adding an attribute to > md patterns, and then the second patch contains the functional > changes. > > One major difficulty in implementing MVE Tail-Predicated Low > Overhead Loops was the need to transform VPT-predicated insns > in the insn chain into their unpredicated equivalents, like: > `mve_vldrbq_z_<supf><mode> -> mve_vldrbq_<supf><mode>`. > > This requires us to have a deterministic link between two > different patterns in mve.md -- this _could_ be done by > re-ordering the entirety of mve.md such that the patterns are > at some constant icode proximity (e.g. having the _z immediately > after the unpredicated version would mean that to map from the > former to the latter you could use icode-1), but that is a very > messy solution that would lead to complex unknown dependencies > between the ordering of patterns. > > This patch proves an alternative way of doing that: using an insn > attribute to encode the icode of the unpredicated instruction. > > No regressions on arm-none-eabi with an MVE target.
This patch is okay once the second one is approved (we'd want them committed together) Thanks, Kyrill > > Thank you, > Stam Markianos-Wright > > gcc/ChangeLog: > > * config/arm/arm.md (mve_unpredicated_insn): New attribute. > * config/arm/arm.h (MVE_VPT_PREDICATED_INSN_P): New define. > (MVE_VPT_UNPREDICATED_INSN_P): Likewise. > (MVE_VPT_PREDICABLE_INSN_P): Likewise. > * config/arm/vec-common.md (mve_vshlq_<supf><mode>): Add > attribute. > * config/arm/mve.md (arm_vcx1q<a>_p_v16qi): Add attribute. > (arm_vcx1q<a>v16qi): Likewise. > (arm_vcx1qav16qi): Likewise. > (arm_vcx1qv16qi): Likewise. > (arm_vcx2q<a>_p_v16qi): Likewise. > (arm_vcx2q<a>v16qi): Likewise. > (arm_vcx2qav16qi): Likewise. > (arm_vcx2qv16qi): Likewise. > (arm_vcx3q<a>_p_v16qi): Likewise. > (arm_vcx3q<a>v16qi): Likewise. > (arm_vcx3qav16qi): Likewise. > (arm_vcx3qv16qi): Likewise. > (mve_vabavq_<supf><mode>): Likewise. > (mve_vabavq_p_<supf><mode>): Likewise. > (mve_vabdq_<supf><mode>): Likewise. > (mve_vabdq_f<mode>): Likewise. > (mve_vabdq_m_<supf><mode>): Likewise. > (mve_vabdq_m_f<mode>): Likewise. > (mve_vabsq_f<mode>): Likewise. > (mve_vabsq_m_f<mode>): Likewise. > (mve_vabsq_m_s<mode>): Likewise. > (mve_vabsq_s<mode>): Likewise. > (mve_vadciq_<supf>v4si): Likewise. > (mve_vadciq_m_<supf>v4si): Likewise. > (mve_vadcq_<supf>v4si): Likewise. > (mve_vadcq_m_<supf>v4si): Likewise. > (mve_vaddlvaq_<supf>v4si): Likewise. > (mve_vaddlvaq_p_<supf>v4si): Likewise. > (mve_vaddlvq_<supf>v4si): Likewise. > (mve_vaddlvq_p_<supf>v4si): Likewise. > (mve_vaddq_f<mode>): Likewise. > (mve_vaddq_m_<supf><mode>): Likewise. > (mve_vaddq_m_f<mode>): Likewise. > (mve_vaddq_m_n_<supf><mode>): Likewise. > (mve_vaddq_m_n_f<mode>): Likewise. > (mve_vaddq_n_<supf><mode>): Likewise. > (mve_vaddq_n_f<mode>): Likewise. > (mve_vaddq<mode>): Likewise. > (mve_vaddvaq_<supf><mode>): Likewise. > (mve_vaddvaq_p_<supf><mode>): Likewise. > (mve_vaddvq_<supf><mode>): Likewise. > (mve_vaddvq_p_<supf><mode>): Likewise. > (mve_vandq_<supf><mode>): Likewise. > (mve_vandq_f<mode>): Likewise. > (mve_vandq_m_<supf><mode>): Likewise. > (mve_vandq_m_f<mode>): Likewise. > (mve_vandq_s<mode>): Likewise. > (mve_vandq_u<mode>): Likewise. > (mve_vbicq_<supf><mode>): Likewise. > (mve_vbicq_f<mode>): Likewise. > (mve_vbicq_m_<supf><mode>): Likewise. > (mve_vbicq_m_f<mode>): Likewise. > (mve_vbicq_m_n_<supf><mode>): Likewise. > (mve_vbicq_n_<supf><mode>): Likewise. > (mve_vbicq_s<mode>): Likewise. > (mve_vbicq_u<mode>): Likewise. > (mve_vbrsrq_m_n_<supf><mode>): Likewise. > (mve_vbrsrq_m_n_f<mode>): Likewise. > (mve_vbrsrq_n_<supf><mode>): Likewise. > (mve_vbrsrq_n_f<mode>): Likewise. > (mve_vcaddq_rot270_m_<supf><mode>): Likewise. > (mve_vcaddq_rot270_m_f<mode>): Likewise. > (mve_vcaddq_rot270<mode>): Likewise. > (mve_vcaddq_rot270<mode>): Likewise. > (mve_vcaddq_rot90_m_<supf><mode>): Likewise. > (mve_vcaddq_rot90_m_f<mode>): Likewise. > (mve_vcaddq_rot90<mode>): Likewise. > (mve_vcaddq_rot90<mode>): Likewise. > (mve_vcaddq<mve_rot><mode>): Likewise. > (mve_vcaddq<mve_rot><mode>): Likewise. > (mve_vclsq_m_s<mode>): Likewise. > (mve_vclsq_s<mode>): Likewise. > (mve_vclzq_<supf><mode>): Likewise. > (mve_vclzq_m_<supf><mode>): Likewise. > (mve_vclzq_s<mode>): Likewise. > (mve_vclzq_u<mode>): Likewise. > (mve_vcmlaq_m_f<mode>): Likewise. > (mve_vcmlaq_rot180_m_f<mode>): Likewise. > (mve_vcmlaq_rot180<mode>): Likewise. > (mve_vcmlaq_rot270_m_f<mode>): Likewise. > (mve_vcmlaq_rot270<mode>): Likewise. > (mve_vcmlaq_rot90_m_f<mode>): Likewise. > (mve_vcmlaq_rot90<mode>): Likewise. > (mve_vcmlaq<mode>): Likewise. > (mve_vcmlaq<mve_rot><mode>): Likewise. > (mve_vcmp<mve_cmp_op>q_<mode>): Likewise. > (mve_vcmp<mve_cmp_op>q_f<mode>): Likewise. > (mve_vcmp<mve_cmp_op>q_n_<mode>): Likewise. > (mve_vcmp<mve_cmp_op>q_n_f<mode>): Likewise. > (mve_vcmpcsq_<mode>): Likewise. > (mve_vcmpcsq_m_n_u<mode>): Likewise. > (mve_vcmpcsq_m_u<mode>): Likewise. > (mve_vcmpcsq_n_<mode>): Likewise. > (mve_vcmpeqq_<mode>): Likewise. > (mve_vcmpeqq_f<mode>): Likewise. > (mve_vcmpeqq_m_<supf><mode>): Likewise. > (mve_vcmpeqq_m_f<mode>): Likewise. > (mve_vcmpeqq_m_n_<supf><mode>): Likewise. > (mve_vcmpeqq_m_n_f<mode>): Likewise. > (mve_vcmpeqq_n_<mode>): Likewise. > (mve_vcmpeqq_n_f<mode>): Likewise. > (mve_vcmpgeq_<mode>): Likewise. > (mve_vcmpgeq_f<mode>): Likewise. > (mve_vcmpgeq_m_f<mode>): Likewise. > (mve_vcmpgeq_m_n_f<mode>): Likewise. > (mve_vcmpgeq_m_n_s<mode>): Likewise. > (mve_vcmpgeq_m_s<mode>): Likewise. > (mve_vcmpgeq_n_<mode>): Likewise. > (mve_vcmpgeq_n_f<mode>): Likewise. > (mve_vcmpgtq_<mode>): Likewise. > (mve_vcmpgtq_f<mode>): Likewise. > (mve_vcmpgtq_m_f<mode>): Likewise. > (mve_vcmpgtq_m_n_f<mode>): Likewise. > (mve_vcmpgtq_m_n_s<mode>): Likewise. > (mve_vcmpgtq_m_s<mode>): Likewise. > (mve_vcmpgtq_n_<mode>): Likewise. > (mve_vcmpgtq_n_f<mode>): Likewise. > (mve_vcmphiq_<mode>): Likewise. > (mve_vcmphiq_m_n_u<mode>): Likewise. > (mve_vcmphiq_m_u<mode>): Likewise. > (mve_vcmphiq_n_<mode>): Likewise. > (mve_vcmpleq_<mode>): Likewise. > (mve_vcmpleq_f<mode>): Likewise. > (mve_vcmpleq_m_f<mode>): Likewise. > (mve_vcmpleq_m_n_f<mode>): Likewise. > (mve_vcmpleq_m_n_s<mode>): Likewise. > (mve_vcmpleq_m_s<mode>): Likewise. > (mve_vcmpleq_n_<mode>): Likewise. > (mve_vcmpleq_n_f<mode>): Likewise. > (mve_vcmpltq_<mode>): Likewise. > (mve_vcmpltq_f<mode>): Likewise. > (mve_vcmpltq_m_f<mode>): Likewise. > (mve_vcmpltq_m_n_f<mode>): Likewise. > (mve_vcmpltq_m_n_s<mode>): Likewise. > (mve_vcmpltq_m_s<mode>): Likewise. > (mve_vcmpltq_n_<mode>): Likewise. > (mve_vcmpltq_n_f<mode>): Likewise. > (mve_vcmpneq_<mode>): Likewise. > (mve_vcmpneq_f<mode>): Likewise. > (mve_vcmpneq_m_<supf><mode>): Likewise. > (mve_vcmpneq_m_f<mode>): Likewise. > (mve_vcmpneq_m_n_<supf><mode>): Likewise. > (mve_vcmpneq_m_n_f<mode>): Likewise. > (mve_vcmpneq_n_<mode>): Likewise. > (mve_vcmpneq_n_f<mode>): Likewise. > (mve_vcmulq_m_f<mode>): Likewise. > (mve_vcmulq_rot180_m_f<mode>): Likewise. > (mve_vcmulq_rot180<mode>): Likewise. > (mve_vcmulq_rot270_m_f<mode>): Likewise. > (mve_vcmulq_rot270<mode>): Likewise. > (mve_vcmulq_rot90_m_f<mode>): Likewise. > (mve_vcmulq_rot90<mode>): Likewise. > (mve_vcmulq<mode>): Likewise. > (mve_vcmulq<mve_rot><mode>): Likewise. > (mve_vctp<mode1>q_mhi): Likewise. > (mve_vctp<mode1>qhi): Likewise. > (mve_vcvtaq_<supf><mode>): Likewise. > (mve_vcvtaq_m_<supf><mode>): Likewise. > (mve_vcvtbq_f16_f32v8hf): Likewise. > (mve_vcvtbq_f32_f16v4sf): Likewise. > (mve_vcvtbq_m_f16_f32v8hf): Likewise. > (mve_vcvtbq_m_f32_f16v4sf): Likewise. > (mve_vcvtmq_<supf><mode>): Likewise. > (mve_vcvtmq_m_<supf><mode>): Likewise. > (mve_vcvtnq_<supf><mode>): Likewise. > (mve_vcvtnq_m_<supf><mode>): Likewise. > (mve_vcvtpq_<supf><mode>): Likewise. > (mve_vcvtpq_m_<supf><mode>): Likewise. > (mve_vcvtq_from_f_<supf><mode>): Likewise. > (mve_vcvtq_m_from_f_<supf><mode>): Likewise. > (mve_vcvtq_m_n_from_f_<supf><mode>): Likewise. > (mve_vcvtq_m_n_to_f_<supf><mode>): Likewise. > (mve_vcvtq_m_to_f_<supf><mode>): Likewise. > (mve_vcvtq_n_from_f_<supf><mode>): Likewise. > (mve_vcvtq_n_to_f_<supf><mode>): Likewise. > (mve_vcvtq_to_f_<supf><mode>): Likewise. > (mve_vcvttq_f16_f32v8hf): Likewise. > (mve_vcvttq_f32_f16v4sf): Likewise. > (mve_vcvttq_m_f16_f32v8hf): Likewise. > (mve_vcvttq_m_f32_f16v4sf): Likewise. > (mve_vddupq_m_wb_u<mode>_insn): Likewise. > (mve_vddupq_u<mode>_insn): Likewise. > (mve_vdupq_m_n_<supf><mode>): Likewise. > (mve_vdupq_m_n_f<mode>): Likewise. > (mve_vdupq_n_<supf><mode>): Likewise. > (mve_vdupq_n_f<mode>): Likewise. > (mve_vdwdupq_m_wb_u<mode>_insn): Likewise. > (mve_vdwdupq_wb_u<mode>_insn): Likewise. > (mve_veorq_<supf><mode>): Likewise. > (mve_veorq_f<mode>): Likewise. > (mve_veorq_m_<supf><mode>): Likewise. > (mve_veorq_m_f<mode>): Likewise. > (mve_veorq_s<mode>): Likewise. > (mve_veorq_u<mode>): Likewise. > (mve_vfmaq_f<mode>): Likewise. > (mve_vfmaq_m_f<mode>): Likewise. > (mve_vfmaq_m_n_f<mode>): Likewise. > (mve_vfmaq_n_f<mode>): Likewise. > (mve_vfmasq_m_n_f<mode>): Likewise. > (mve_vfmasq_n_f<mode>): Likewise. > (mve_vfmsq_f<mode>): Likewise. > (mve_vfmsq_m_f<mode>): Likewise. > (mve_vhaddq_<supf><mode>): Likewise. > (mve_vhaddq_m_<supf><mode>): Likewise. > (mve_vhaddq_m_n_<supf><mode>): Likewise. > (mve_vhaddq_n_<supf><mode>): Likewise. > (mve_vhcaddq_rot270_m_s<mode>): Likewise. > (mve_vhcaddq_rot270_s<mode>): Likewise. > (mve_vhcaddq_rot90_m_s<mode>): Likewise. > (mve_vhcaddq_rot90_s<mode>): Likewise. > (mve_vhsubq_<supf><mode>): Likewise. > (mve_vhsubq_m_<supf><mode>): Likewise. > (mve_vhsubq_m_n_<supf><mode>): Likewise. > (mve_vhsubq_n_<supf><mode>): Likewise. > (mve_vidupq_m_wb_u<mode>_insn): Likewise. > (mve_vidupq_u<mode>_insn): Likewise. > (mve_viwdupq_m_wb_u<mode>_insn): Likewise. > (mve_viwdupq_wb_u<mode>_insn): Likewise. > (mve_vldrbq_<supf><mode>): Likewise. > (mve_vldrbq_gather_offset_<supf><mode>): Likewise. > (mve_vldrbq_gather_offset_z_<supf><mode>): Likewise. > (mve_vldrbq_z_<supf><mode>): Likewise. > (mve_vldrdq_gather_base_<supf>v2di): Likewise. > (mve_vldrdq_gather_base_wb_<supf>v2di_insn): Likewise. > (mve_vldrdq_gather_base_wb_z_<supf>v2di_insn): Likewise. > (mve_vldrdq_gather_base_z_<supf>v2di): Likewise. > (mve_vldrdq_gather_offset_<supf>v2di): Likewise. > (mve_vldrdq_gather_offset_z_<supf>v2di): Likewise. > (mve_vldrdq_gather_shifted_offset_<supf>v2di): Likewise. > (mve_vldrdq_gather_shifted_offset_z_<supf>v2di): Likewise. > (mve_vldrhq_<supf><mode>): Likewise. > (mve_vldrhq_fv8hf): Likewise. > (mve_vldrhq_gather_offset_<supf><mode>): Likewise. > (mve_vldrhq_gather_offset_fv8hf): Likewise. > (mve_vldrhq_gather_offset_z_<supf><mode>): Likewise. > (mve_vldrhq_gather_offset_z_fv8hf): Likewise. > (mve_vldrhq_gather_shifted_offset_<supf><mode>): Likewise. > (mve_vldrhq_gather_shifted_offset_fv8hf): Likewise. > (mve_vldrhq_gather_shifted_offset_z_<supf><mode>): Likewise. > (mve_vldrhq_gather_shifted_offset_z_fv8hf): Likewise. > (mve_vldrhq_z_<supf><mode>): Likewise. > (mve_vldrhq_z_fv8hf): Likewise. > (mve_vldrwq_<supf>v4si): Likewise. > (mve_vldrwq_fv4sf): Likewise. > (mve_vldrwq_gather_base_<supf>v4si): Likewise. > (mve_vldrwq_gather_base_fv4sf): Likewise. > (mve_vldrwq_gather_base_wb_<supf>v4si_insn): Likewise. > (mve_vldrwq_gather_base_wb_fv4sf_insn): Likewise. > (mve_vldrwq_gather_base_wb_z_<supf>v4si_insn): Likewise. > (mve_vldrwq_gather_base_wb_z_fv4sf_insn): Likewise. > (mve_vldrwq_gather_base_z_<supf>v4si): Likewise. > (mve_vldrwq_gather_base_z_fv4sf): Likewise. > (mve_vldrwq_gather_offset_<supf>v4si): Likewise. > (mve_vldrwq_gather_offset_fv4sf): Likewise. > (mve_vldrwq_gather_offset_z_<supf>v4si): Likewise. > (mve_vldrwq_gather_offset_z_fv4sf): Likewise. > (mve_vldrwq_gather_shifted_offset_<supf>v4si): Likewise. > (mve_vldrwq_gather_shifted_offset_fv4sf): Likewise. > (mve_vldrwq_gather_shifted_offset_z_<supf>v4si): Likewise. > (mve_vldrwq_gather_shifted_offset_z_fv4sf): Likewise. > (mve_vldrwq_z_<supf>v4si): Likewise. > (mve_vldrwq_z_fv4sf): Likewise. > (mve_vmaxaq_m_s<mode>): Likewise. > (mve_vmaxaq_s<mode>): Likewise. > (mve_vmaxavq_p_s<mode>): Likewise. > (mve_vmaxavq_s<mode>): Likewise. > (mve_vmaxnmaq_f<mode>): Likewise. > (mve_vmaxnmaq_m_f<mode>): Likewise. > (mve_vmaxnmavq_f<mode>): Likewise. > (mve_vmaxnmavq_p_f<mode>): Likewise. > (mve_vmaxnmq_f<mode>): Likewise. > (mve_vmaxnmq_m_f<mode>): Likewise. > (mve_vmaxnmvq_f<mode>): Likewise. > (mve_vmaxnmvq_p_f<mode>): Likewise. > (mve_vmaxq_<supf><mode>): Likewise. > (mve_vmaxq_m_<supf><mode>): Likewise. > (mve_vmaxq_s<mode>): Likewise. > (mve_vmaxq_u<mode>): Likewise. > (mve_vmaxvq_<supf><mode>): Likewise. > (mve_vmaxvq_p_<supf><mode>): Likewise. > (mve_vminaq_m_s<mode>): Likewise. > (mve_vminaq_s<mode>): Likewise. > (mve_vminavq_p_s<mode>): Likewise. > (mve_vminavq_s<mode>): Likewise. > (mve_vminnmaq_f<mode>): Likewise. > (mve_vminnmaq_m_f<mode>): Likewise. > (mve_vminnmavq_f<mode>): Likewise. > (mve_vminnmavq_p_f<mode>): Likewise. > (mve_vminnmq_f<mode>): Likewise. > (mve_vminnmq_m_f<mode>): Likewise. > (mve_vminnmvq_f<mode>): Likewise. > (mve_vminnmvq_p_f<mode>): Likewise. > (mve_vminq_<supf><mode>): Likewise. > (mve_vminq_m_<supf><mode>): Likewise. > (mve_vminq_s<mode>): Likewise. > (mve_vminq_u<mode>): Likewise. > (mve_vminvq_<supf><mode>): Likewise. > (mve_vminvq_p_<supf><mode>): Likewise. > (mve_vmladavaq_<supf><mode>): Likewise. > (mve_vmladavaq_p_<supf><mode>): Likewise. > (mve_vmladavaxq_p_s<mode>): Likewise. > (mve_vmladavaxq_s<mode>): Likewise. > (mve_vmladavq_<supf><mode>): Likewise. > (mve_vmladavq_p_<supf><mode>): Likewise. > (mve_vmladavxq_p_s<mode>): Likewise. > (mve_vmladavxq_s<mode>): Likewise. > (mve_vmlaldavaq_<supf><mode>): Likewise. > (mve_vmlaldavaq_p_<supf><mode>): Likewise. > (mve_vmlaldavaxq_<supf><mode>): Likewise. > (mve_vmlaldavaxq_p_<supf><mode>): Likewise. > (mve_vmlaldavaxq_s<mode>): Likewise. > (mve_vmlaldavq_<supf><mode>): Likewise. > (mve_vmlaldavq_p_<supf><mode>): Likewise. > (mve_vmlaldavxq_p_s<mode>): Likewise. > (mve_vmlaldavxq_s<mode>): Likewise. > (mve_vmlaq_m_n_<supf><mode>): Likewise. > (mve_vmlaq_n_<supf><mode>): Likewise. > (mve_vmlasq_m_n_<supf><mode>): Likewise. > (mve_vmlasq_n_<supf><mode>): Likewise. > (mve_vmlsdavaq_p_s<mode>): Likewise. > (mve_vmlsdavaq_s<mode>): Likewise. > (mve_vmlsdavaxq_p_s<mode>): Likewise. > (mve_vmlsdavaxq_s<mode>): Likewise. > (mve_vmlsdavq_p_s<mode>): Likewise. > (mve_vmlsdavq_s<mode>): Likewise. > (mve_vmlsdavxq_p_s<mode>): Likewise. > (mve_vmlsdavxq_s<mode>): Likewise. > (mve_vmlsldavaq_p_s<mode>): Likewise. > (mve_vmlsldavaq_s<mode>): Likewise. > (mve_vmlsldavaxq_p_s<mode>): Likewise. > (mve_vmlsldavaxq_s<mode>): Likewise. > (mve_vmlsldavq_p_s<mode>): Likewise. > (mve_vmlsldavq_s<mode>): Likewise. > (mve_vmlsldavxq_p_s<mode>): Likewise. > (mve_vmlsldavxq_s<mode>): Likewise. > (mve_vmovlbq_<supf><mode>): Likewise. > (mve_vmovlbq_m_<supf><mode>): Likewise. > (mve_vmovltq_<supf><mode>): Likewise. > (mve_vmovltq_m_<supf><mode>): Likewise. > (mve_vmovnbq_<supf><mode>): Likewise. > (mve_vmovnbq_m_<supf><mode>): Likewise. > (mve_vmovntq_<supf><mode>): Likewise. > (mve_vmovntq_m_<supf><mode>): Likewise. > (mve_vmulhq_<supf><mode>): Likewise. > (mve_vmulhq_m_<supf><mode>): Likewise. > (mve_vmullbq_int_<supf><mode>): Likewise. > (mve_vmullbq_int_m_<supf><mode>): Likewise. > (mve_vmullbq_poly_m_p<mode>): Likewise. > (mve_vmullbq_poly_p<mode>): Likewise. > (mve_vmulltq_int_<supf><mode>): Likewise. > (mve_vmulltq_int_m_<supf><mode>): Likewise. > (mve_vmulltq_poly_m_p<mode>): Likewise. > (mve_vmulltq_poly_p<mode>): Likewise. > (mve_vmulq_<supf><mode>): Likewise. > (mve_vmulq_f<mode>): Likewise. > (mve_vmulq_m_<supf><mode>): Likewise. > (mve_vmulq_m_f<mode>): Likewise. > (mve_vmulq_m_n_<supf><mode>): Likewise. > (mve_vmulq_m_n_f<mode>): Likewise. > (mve_vmulq_n_<supf><mode>): Likewise. > (mve_vmulq_n_f<mode>): Likewise. > (mve_vmvnq_<supf><mode>): Likewise. > (mve_vmvnq_m_<supf><mode>): Likewise. > (mve_vmvnq_m_n_<supf><mode>): Likewise. > (mve_vmvnq_n_<supf><mode>): Likewise. > (mve_vmvnq_s<mode>): Likewise. > (mve_vmvnq_u<mode>): Likewise. > (mve_vnegq_f<mode>): Likewise. > (mve_vnegq_m_f<mode>): Likewise. > (mve_vnegq_m_s<mode>): Likewise. > (mve_vnegq_s<mode>): Likewise. > (mve_vornq_<supf><mode>): Likewise. > (mve_vornq_f<mode>): Likewise. > (mve_vornq_m_<supf><mode>): Likewise. > (mve_vornq_m_f<mode>): Likewise. > (mve_vornq_s<mode>): Likewise. > (mve_vornq_u<mode>): Likewise. > (mve_vorrq_<supf><mode>): Likewise. > (mve_vorrq_f<mode>): Likewise. > (mve_vorrq_m_<supf><mode>): Likewise. > (mve_vorrq_m_f<mode>): Likewise. > (mve_vorrq_m_n_<supf><mode>): Likewise. > (mve_vorrq_n_<supf><mode>): Likewise. > (mve_vorrq_s<mode>): Likewise. > (mve_vorrq_s<mode>): Likewise. > (mve_vqabsq_m_s<mode>): Likewise. > (mve_vqabsq_s<mode>): Likewise. > (mve_vqaddq_<supf><mode>): Likewise. > (mve_vqaddq_m_<supf><mode>): Likewise. > (mve_vqaddq_m_n_<supf><mode>): Likewise. > (mve_vqaddq_n_<supf><mode>): Likewise. > (mve_vqdmladhq_m_s<mode>): Likewise. > (mve_vqdmladhq_s<mode>): Likewise. > (mve_vqdmladhxq_m_s<mode>): Likewise. > (mve_vqdmladhxq_s<mode>): Likewise. > (mve_vqdmlahq_m_n_s<mode>): Likewise. > (mve_vqdmlahq_n_<supf><mode>): Likewise. > (mve_vqdmlahq_n_s<mode>): Likewise. > (mve_vqdmlashq_m_n_s<mode>): Likewise. > (mve_vqdmlashq_n_<supf><mode>): Likewise. > (mve_vqdmlashq_n_s<mode>): Likewise. > (mve_vqdmlsdhq_m_s<mode>): Likewise. > (mve_vqdmlsdhq_s<mode>): Likewise. > (mve_vqdmlsdhxq_m_s<mode>): Likewise. > (mve_vqdmlsdhxq_s<mode>): Likewise. > (mve_vqdmulhq_m_n_s<mode>): Likewise. > (mve_vqdmulhq_m_s<mode>): Likewise. > (mve_vqdmulhq_n_s<mode>): Likewise. > (mve_vqdmulhq_s<mode>): Likewise. > (mve_vqdmullbq_m_n_s<mode>): Likewise. > (mve_vqdmullbq_m_s<mode>): Likewise. > (mve_vqdmullbq_n_s<mode>): Likewise. > (mve_vqdmullbq_s<mode>): Likewise. > (mve_vqdmulltq_m_n_s<mode>): Likewise. > (mve_vqdmulltq_m_s<mode>): Likewise. > (mve_vqdmulltq_n_s<mode>): Likewise. > (mve_vqdmulltq_s<mode>): Likewise. > (mve_vqmovnbq_<supf><mode>): Likewise. > (mve_vqmovnbq_m_<supf><mode>): Likewise. > (mve_vqmovntq_<supf><mode>): Likewise. > (mve_vqmovntq_m_<supf><mode>): Likewise. > (mve_vqmovunbq_m_s<mode>): Likewise. > (mve_vqmovunbq_s<mode>): Likewise. > (mve_vqmovuntq_m_s<mode>): Likewise. > (mve_vqmovuntq_s<mode>): Likewise. > (mve_vqnegq_m_s<mode>): Likewise. > (mve_vqnegq_s<mode>): Likewise. > (mve_vqrdmladhq_m_s<mode>): Likewise. > (mve_vqrdmladhq_s<mode>): Likewise. > (mve_vqrdmladhxq_m_s<mode>): Likewise. > (mve_vqrdmladhxq_s<mode>): Likewise. > (mve_vqrdmlahq_m_n_s<mode>): Likewise. > (mve_vqrdmlahq_n_<supf><mode>): Likewise. > (mve_vqrdmlahq_n_s<mode>): Likewise. > (mve_vqrdmlashq_m_n_s<mode>): Likewise. > (mve_vqrdmlashq_n_<supf><mode>): Likewise. > (mve_vqrdmlashq_n_s<mode>): Likewise. > (mve_vqrdmlsdhq_m_s<mode>): Likewise. > (mve_vqrdmlsdhq_s<mode>): Likewise. > (mve_vqrdmlsdhxq_m_s<mode>): Likewise. > (mve_vqrdmlsdhxq_s<mode>): Likewise. > (mve_vqrdmulhq_m_n_s<mode>): Likewise. > (mve_vqrdmulhq_m_s<mode>): Likewise. > (mve_vqrdmulhq_n_s<mode>): Likewise. > (mve_vqrdmulhq_s<mode>): Likewise. > (mve_vqrshlq_<supf><mode>): Likewise. > (mve_vqrshlq_m_<supf><mode>): Likewise. > (mve_vqrshlq_m_n_<supf><mode>): Likewise. > (mve_vqrshlq_n_<supf><mode>): Likewise. > (mve_vqrshrnbq_m_n_<supf><mode>): Likewise. > (mve_vqrshrnbq_n_<supf><mode>): Likewise. > (mve_vqrshrntq_m_n_<supf><mode>): Likewise. > (mve_vqrshrntq_n_<supf><mode>): Likewise. > (mve_vqrshrunbq_m_n_s<mode>): Likewise. > (mve_vqrshrunbq_n_s<mode>): Likewise. > (mve_vqrshruntq_m_n_s<mode>): Likewise. > (mve_vqrshruntq_n_s<mode>): Likewise. > (mve_vqshlq_<supf><mode>): Likewise. > (mve_vqshlq_m_<supf><mode>): Likewise. > (mve_vqshlq_m_n_<supf><mode>): Likewise. > (mve_vqshlq_m_r_<supf><mode>): Likewise. > (mve_vqshlq_n_<supf><mode>): Likewise. > (mve_vqshlq_r_<supf><mode>): Likewise. > (mve_vqshluq_m_n_s<mode>): Likewise. > (mve_vqshluq_n_s<mode>): Likewise. > (mve_vqshrnbq_m_n_<supf><mode>): Likewise. > (mve_vqshrnbq_n_<supf><mode>): Likewise. > (mve_vqshrntq_m_n_<supf><mode>): Likewise. > (mve_vqshrntq_n_<supf><mode>): Likewise. > (mve_vqshrunbq_m_n_s<mode>): Likewise. > (mve_vqshrunbq_n_s<mode>): Likewise. > (mve_vqshruntq_m_n_s<mode>): Likewise. > (mve_vqshruntq_n_s<mode>): Likewise. > (mve_vqsubq_<supf><mode>): Likewise. > (mve_vqsubq_m_<supf><mode>): Likewise. > (mve_vqsubq_m_n_<supf><mode>): Likewise. > (mve_vqsubq_n_<supf><mode>): Likewise. > (mve_vrev16q_<supf>v16qi): Likewise. > (mve_vrev16q_m_<supf>v16qi): Likewise. > (mve_vrev32q_<supf><mode>): Likewise. > (mve_vrev32q_fv8hf): Likewise. > (mve_vrev32q_m_<supf><mode>): Likewise. > (mve_vrev32q_m_fv8hf): Likewise. > (mve_vrev64q_<supf><mode>): Likewise. > (mve_vrev64q_f<mode>): Likewise. > (mve_vrev64q_m_<supf><mode>): Likewise. > (mve_vrev64q_m_f<mode>): Likewise. > (mve_vrhaddq_<supf><mode>): Likewise. > (mve_vrhaddq_m_<supf><mode>): Likewise. > (mve_vrmlaldavhaq_<supf>v4si): Likewise. > (mve_vrmlaldavhaq_p_sv4si): Likewise. > (mve_vrmlaldavhaq_p_uv4si): Likewise. > (mve_vrmlaldavhaq_sv4si): Likewise. > (mve_vrmlaldavhaq_uv4si): Likewise. > (mve_vrmlaldavhaxq_p_sv4si): Likewise. > (mve_vrmlaldavhaxq_sv4si): Likewise. > (mve_vrmlaldavhq_<supf>v4si): Likewise. > (mve_vrmlaldavhq_p_<supf>v4si): Likewise. > (mve_vrmlaldavhxq_p_sv4si): Likewise. > (mve_vrmlaldavhxq_sv4si): Likewise. > (mve_vrmlsldavhaq_p_sv4si): Likewise. > (mve_vrmlsldavhaq_sv4si): Likewise. > (mve_vrmlsldavhaxq_p_sv4si): Likewise. > (mve_vrmlsldavhaxq_sv4si): Likewise. > (mve_vrmlsldavhq_p_sv4si): Likewise. > (mve_vrmlsldavhq_sv4si): Likewise. > (mve_vrmlsldavhxq_p_sv4si): Likewise. > (mve_vrmlsldavhxq_sv4si): Likewise. > (mve_vrmulhq_<supf><mode>): Likewise. > (mve_vrmulhq_m_<supf><mode>): Likewise. > (mve_vrndaq_f<mode>): Likewise. > (mve_vrndaq_m_f<mode>): Likewise. > (mve_vrndmq_f<mode>): Likewise. > (mve_vrndmq_m_f<mode>): Likewise. > (mve_vrndnq_f<mode>): Likewise. > (mve_vrndnq_m_f<mode>): Likewise. > (mve_vrndpq_f<mode>): Likewise. > (mve_vrndpq_m_f<mode>): Likewise. > (mve_vrndq_f<mode>): Likewise. > (mve_vrndq_m_f<mode>): Likewise. > (mve_vrndxq_f<mode>): Likewise. > (mve_vrndxq_m_f<mode>): Likewise. > (mve_vrshlq_<supf><mode>): Likewise. > (mve_vrshlq_m_<supf><mode>): Likewise. > (mve_vrshlq_m_n_<supf><mode>): Likewise. > (mve_vrshlq_n_<supf><mode>): Likewise. > (mve_vrshrnbq_m_n_<supf><mode>): Likewise. > (mve_vrshrnbq_n_<supf><mode>): Likewise. > (mve_vrshrntq_m_n_<supf><mode>): Likewise. > (mve_vrshrntq_n_<supf><mode>): Likewise. > (mve_vrshrq_m_n_<supf><mode>): Likewise. > (mve_vrshrq_n_<supf><mode>): Likewise. > (mve_vsbciq_<supf>v4si): Likewise. > (mve_vsbciq_m_<supf>v4si): Likewise. > (mve_vsbcq_<supf>v4si): Likewise. > (mve_vsbcq_m_<supf>v4si): Likewise. > (mve_vshlcq_<supf><mode>): Likewise. > (mve_vshlcq_m_<supf><mode>): Likewise. > (mve_vshllbq_m_n_<supf><mode>): Likewise. > (mve_vshllbq_n_<supf><mode>): Likewise. > (mve_vshlltq_m_n_<supf><mode>): Likewise. > (mve_vshlltq_n_<supf><mode>): Likewise. > (mve_vshlq_<supf><mode>): Likewise. > (mve_vshlq_<supf><mode>): Likewise. > (mve_vshlq_m_<supf><mode>): Likewise. > (mve_vshlq_m_n_<supf><mode>): Likewise. > (mve_vshlq_m_r_<supf><mode>): Likewise. > (mve_vshlq_n_<supf><mode>): Likewise. > (mve_vshlq_r_<supf><mode>): Likewise. > (mve_vshrnbq_m_n_<supf><mode>): Likewise. > (mve_vshrnbq_n_<supf><mode>): Likewise. > (mve_vshrntq_m_n_<supf><mode>): Likewise. > (mve_vshrntq_n_<supf><mode>): Likewise. > (mve_vshrq_m_n_<supf><mode>): Likewise. > (mve_vshrq_n_<supf><mode>): Likewise. > (mve_vsliq_m_n_<supf><mode>): Likewise. > (mve_vsliq_n_<supf><mode>): Likewise. > (mve_vsriq_m_n_<supf><mode>): Likewise. > (mve_vsriq_n_<supf><mode>): Likewise. > (mve_vstrbq_<supf><mode>): Likewise. > (mve_vstrbq_p_<supf><mode>): Likewise. > (mve_vstrbq_scatter_offset_<supf><mode>_insn): Likewise. > (mve_vstrbq_scatter_offset_p_<supf><mode>_insn): Likewise. > (mve_vstrdq_scatter_base_<supf>v2di): Likewise. > (mve_vstrdq_scatter_base_p_<supf>v2di): Likewise. > (mve_vstrdq_scatter_base_wb_<supf>v2di): Likewise. > (mve_vstrdq_scatter_base_wb_p_<supf>v2di): Likewise. > (mve_vstrdq_scatter_offset_<supf>v2di_insn): Likewise. > (mve_vstrdq_scatter_offset_p_<supf>v2di_insn): Likewise. > (mve_vstrdq_scatter_shifted_offset_<supf>v2di_insn): Likewise. > (mve_vstrdq_scatter_shifted_offset_p_<supf>v2di_insn): Likewise. > (mve_vstrhq_<supf><mode>): Likewise. > (mve_vstrhq_fv8hf): Likewise. > (mve_vstrhq_p_<supf><mode>): Likewise. > (mve_vstrhq_p_fv8hf): Likewise. > (mve_vstrhq_scatter_offset_<supf><mode>_insn): Likewise. > (mve_vstrhq_scatter_offset_fv8hf_insn): Likewise. > (mve_vstrhq_scatter_offset_p_<supf><mode>_insn): Likewise. > (mve_vstrhq_scatter_offset_p_fv8hf_insn): Likewise. > (mve_vstrhq_scatter_shifted_offset_<supf><mode>_insn): Likewise. > (mve_vstrhq_scatter_shifted_offset_fv8hf_insn): Likewise. > (mve_vstrhq_scatter_shifted_offset_p_<supf><mode>_insn): Likewise. > (mve_vstrhq_scatter_shifted_offset_p_fv8hf_insn): Likewise. > (mve_vstrwq_<supf>v4si): Likewise. > (mve_vstrwq_fv4sf): Likewise. > (mve_vstrwq_p_<supf>v4si): Likewise. > (mve_vstrwq_p_fv4sf): Likewise. > (mve_vstrwq_scatter_base_<supf>v4si): Likewise. > (mve_vstrwq_scatter_base_fv4sf): Likewise. > (mve_vstrwq_scatter_base_p_<supf>v4si): Likewise. > (mve_vstrwq_scatter_base_p_fv4sf): Likewise. > (mve_vstrwq_scatter_base_wb_<supf>v4si): Likewise. > (mve_vstrwq_scatter_base_wb_fv4sf): Likewise. > (mve_vstrwq_scatter_base_wb_p_<supf>v4si): Likewise. > (mve_vstrwq_scatter_base_wb_p_fv4sf): Likewise. > (mve_vstrwq_scatter_offset_<supf>v4si_insn): Likewise. > (mve_vstrwq_scatter_offset_fv4sf_insn): Likewise. > (mve_vstrwq_scatter_offset_p_<supf>v4si_insn): Likewise. > (mve_vstrwq_scatter_offset_p_fv4sf_insn): Likewise. > (mve_vstrwq_scatter_shifted_offset_<supf>v4si_insn): Likewise. > (mve_vstrwq_scatter_shifted_offset_fv4sf_insn): Likewise. > (mve_vstrwq_scatter_shifted_offset_p_<supf>v4si_insn): Likewise. > (mve_vstrwq_scatter_shifted_offset_p_fv4sf_insn): Likewise. > (mve_vsubq_<supf><mode>): Likewise. > (mve_vsubq_f<mode>): Likewise. > (mve_vsubq_m_<supf><mode>): Likewise. > (mve_vsubq_m_f<mode>): Likewise. > (mve_vsubq_m_n_<supf><mode>): Likewise. > (mve_vsubq_m_n_f<mode>): Likewise. > (mve_vsubq_n_<supf><mode>): Likewise. > (mve_vsubq_n_f<mode>): Likewise.
