LGTM, Thanks, it's really awesome - the implementation is simpler than I expected, it's another great improvement for RISC-V GCC!
Just make sure Patrick gives a green light on the testing before committing the patch :) On Wed, Oct 25, 2023 at 8:05 PM Juzhe-Zhong <juzhe.zh...@rivai.ai> wrote: > > This patch addresses the redundant AVL/VL toggling in RVV partial > auto-vectorization > which is a known issue for a long time and I finally find the time to address > it. > > Consider a simple vector addition operation: > > https://godbolt.org/z/7hfGfEjW3 > > void > foo (int *__restrict a, > int *__restrict b, > int *__restrict n) > { > for (int i = 0; i < n; i++) > a[i] = a[i] + b[i]; > } > > Optimized IR: > > Loop body: > _38 = .SELECT_VL (ivtmp_36, POLY_INT_CST [4, 4]); > -> vsetvli a5,a2,e8,mf4,ta,ma > ... > vect__4.8_27 = .MASK_LEN_LOAD (vectp_a.6_29, 32B, { -1, ... }, _38, 0); > -> vle32.v v2,0(a0) > vect__6.11_20 = .MASK_LEN_LOAD (vectp_b.9_25, 32B, { -1, ... }, _38, 0); > -> vle32.v v1,0(a1) > vect__7.12_19 = vect__6.11_20 + vect__4.8_27; > -> vsetvli a6,zero,e32,m1,ta,ma + vadd.vv v1,v1,v2 > .MASK_LEN_STORE (vectp_a.13_11, 32B, { -1, ... }, _38, 0, vect__7.12_19); > -> vsetvli zero,a5,e32,m1,ta,ma + vse32.v v1,0(a4) > > We can see 2 redundant vsetvls inside the loop body due to AVL/VL toggling. > The AVL/VL toggling is because we are missing LEN information in simple > PLUS_EXPR GIMPLE assignment: > > vect__7.12_19 = vect__6.11_20 + vect__4.8_27; > > GCC apply partial predicate load/store and un-predicated full vector > operation on partial vectorization. > Such flow are used by all other targets like ARM SVE (RVV also uses such > flow): > > ARM SVE: > > .L3: > ld1w z30.s, p7/z, [x0, x3, lsl 2] -> predicated load > ld1w z31.s, p7/z, [x1, x3, lsl 2] -> predicated load > add z31.s, z31.s, z30.s -> un-predicated add > st1w z31.s, p7, [x0, x3, lsl 2] -> predicated store > > Such vectorization flow causes AVL/VL toggling on RVV so we need AVL > propagation PASS for it. > > Also, It's very unlikely that we can apply predicated operations on all > vectorization for following reasons: > > 1. It's very heavy workload to support them on all vectorization and we don't > see any benefits if we can handle that on targets backend. > 2. Changing Loop vectorizer for it will make code base ugly and hard to > maintain. > 3. We will need so many patterns for all operations. Not only COND_LEN_ADD, > COND_LEN_SUB, .... > We also need COND_LEN_EXTEND, ...., COND_LEN_CEIL, ... .. over 100+ > patterns, unreasonable number of patterns. > > To conclude, we prefer un-predicated operations here, and design a nice and > clean AVL propagation PASS for it to elide the redundant vsetvls > due to AVL/VL toggling. > > The second question is that why we separate a PASS called AVL propagation. > Why not optimize it in VSETVL PASS (We definitetly can optimize AVL in VSETVL > PASS) > > Frankly, I was planning to address such issue in VSETVL PASS that's why we > recently refactored VSETVL PASS. However, I changed my mind recently after > several > experiments and tries. > > The reasons as follows: > > 1. For code base management and maintainience. Current VSETVL PASS is > complicated enough and aleady has enough aggressive and fancy optimizations > which > turns out it can always generate optimal codegen in most of the cases. > It's not a good idea keep adding more features into VSETVL PASS to make VSETVL > PASS become heavy and heavy again, then we will need to refactor it > again in the future. > Actuall, the VSETVL PASS is very stable and optimal after the recent > refactoring. Hopefully, we should not change VSETVL PASS any more except the > minor > fixes. > > 2. vsetvl insertion (VSETVL PASS does this thing) and AVL propagation are 2 > different things, I don't think we should fuse them into same PASS. > > 3. VSETVL PASS is an post-RA PASS, wheras AVL propagtion should be done > before RA which can reduce register allocation. > > 4. This patch's AVL propagation PASS only does AVL propagation for RVV > partial auto-vectorization situations. > This patch's codes are only hundreds lines which is very managable and can > be very easily extended features and enhancements. > We can easily extend and enhance more AVL propagation in a clean and > separate PASS in the future. (If we do it on VSETVL PASS, we will complicate > VSETVL PASS again which is already so complicated.) > > Here is an example to demonstrate more: > > https://godbolt.org/z/bE86sv3q5 > > void foo2 (int *__restrict a, > int *__restrict b, > int *__restrict c, > int *__restrict a2, > int *__restrict b2, > int *__restrict c2, > int *__restrict a3, > int *__restrict b3, > int *__restrict c3, > int *__restrict a4, > int *__restrict b4, > int *__restrict c4, > int *__restrict a5, > int *__restrict b5, > int *__restrict c5, > int n) > { > for (int i = 0; i < n; i++){ > a[i] = b[i] + c[i]; > b5[i] = b[i] + c[i]; > a2[i] = b2[i] + c2[i]; > a3[i] = b3[i] + c3[i]; > a4[i] = b4[i] + c4[i]; > a5[i] = a[i] + a4[i]; > a[i] = a5[i] + b5[i]+ a[i]; > > a[i] = a[i] + c[i]; > b5[i] = a[i] + c[i]; > a2[i] = a[i] + c2[i]; > a3[i] = a[i] + c3[i]; > a4[i] = a[i] + c4[i]; > a5[i] = a[i] + a4[i]; > a[i] = a[i] + b5[i]+ a[i]; > } > } > > 1. Loop Body: > > Before this patch: After this patch: > > vsetvli a4,t1,e8,mf4,ta,ma vsetvli > a4,t1,e32,m1,ta,ma > vle32.v v2,0(a2) vle32.v > v2,0(a2) > vle32.v v4,0(a1) vle32.v > v3,0(t2) > vle32.v v1,0(t2) vle32.v > v4,0(a1) > vsetvli a7,zero,e32,m1,ta,ma vle32.v > v1,0(t0) > vadd.vv v4,v2,v4 vadd.vv > v4,v2,v4 > vsetvli zero,a4,e32,m1,ta,ma vadd.vv > v1,v3,v1 > vle32.v v3,0(s0) vadd.vv > v1,v1,v4 > vsetvli a7,zero,e32,m1,ta,ma vadd.vv > v1,v1,v4 > vadd.vv v1,v3,v1 vadd.vv > v1,v1,v4 > vadd.vv v1,v1,v4 vadd.vv > v1,v1,v2 > vadd.vv v1,v1,v4 vadd.vv > v2,v1,v2 > vadd.vv v1,v1,v4 vse32.v > v2,0(t5) > vsetvli zero,a4,e32,m1,ta,ma vadd.vv > v2,v2,v1 > vle32.v v4,0(a5) vadd.vv > v2,v2,v1 > vsetvli a7,zero,e32,m1,ta,ma slli > a7,a4,2 > vadd.vv v1,v1,v2 vadd.vv > v3,v1,v3 > vadd.vv v2,v1,v2 vle32.v > v5,0(a5) > vadd.vv v4,v1,v4 vle32.v > v6,0(t6) > vsetvli zero,a4,e32,m1,ta,ma vse32.v > v3,0(t3) > vse32.v v2,0(t5) vse32.v > v2,0(a0) > vse32.v v4,0(a3) vadd.vv > v3,v3,v1 > vsetvli a7,zero,e32,m1,ta,ma vadd.vv > v2,v1,v5 > vadd.vv v3,v1,v3 vse32.v > v3,0(t4) > vadd.vv v2,v2,v1 vadd.vv > v1,v1,v6 > vadd.vv v2,v2,v1 vse32.v > v2,0(a3) > vsetvli zero,a4,e32,m1,ta,ma vse32.v > v1,0(a6) > vse32.v v2,0(a0) > vse32.v v3,0(t3) > vle32.v v2,0(t0) > vsetvli a7,zero,e32,m1,ta,ma > vadd.vv v3,v3,v1 > vsetvli zero,a4,e32,m1,ta,ma > vse32.v v3,0(t4) > vsetvli a7,zero,e32,m1,ta,ma > slli a7,a4,2 > vadd.vv v1,v1,v2 > sub t1,t1,a4 > vsetvli zero,a4,e32,m1,ta,ma > vse32.v v1,0(a6) > > It's quite obvious, all heavy && redundant vsetvls inside loop body are > eliminated. > > 2. Epilogue: > Before this patch: After this > patch: > > .L5: .L5: > ld s0,8(sp) ret > addi sp,sp,16 > jr ra > > This is the benefit we do the AVL propation before RA since we eliminate the > use of 'a7' register > which is used by the redudant AVL/VL toggling instruction: 'vsetvli > a7,zero,e32,m1,ta,ma' > > The final codegen after this patch: > > foo2: > lw t1,56(sp) > ld t6,0(sp) > ld t3,8(sp) > ld t0,16(sp) > ld t2,24(sp) > ld t4,32(sp) > ld t5,40(sp) > ble t1,zero,.L5 > .L3: > vsetvli a4,t1,e32,m1,ta,ma > vle32.v v2,0(a2) > vle32.v v3,0(t2) > vle32.v v4,0(a1) > vle32.v v1,0(t0) > vadd.vv v4,v2,v4 > vadd.vv v1,v3,v1 > vadd.vv v1,v1,v4 > vadd.vv v1,v1,v4 > vadd.vv v1,v1,v4 > vadd.vv v1,v1,v2 > vadd.vv v2,v1,v2 > vse32.v v2,0(t5) > vadd.vv v2,v2,v1 > vadd.vv v2,v2,v1 > slli a7,a4,2 > vadd.vv v3,v1,v3 > vle32.v v5,0(a5) > vle32.v v6,0(t6) > vse32.v v3,0(t3) > vse32.v v2,0(a0) > vadd.vv v3,v3,v1 > vadd.vv v2,v1,v5 > vse32.v v3,0(t4) > vadd.vv v1,v1,v6 > vse32.v v2,0(a3) > vse32.v v1,0(a6) > sub t1,t1,a4 > add a1,a1,a7 > add a2,a2,a7 > add a5,a5,a7 > add t6,t6,a7 > add t0,t0,a7 > add t2,t2,a7 > add t5,t5,a7 > add a3,a3,a7 > add a6,a6,a7 > add t3,t3,a7 > add t4,t4,a7 > add a0,a0,a7 > bne t1,zero,.L3 > .L5: > ret > > > PR target/111318 > PR target/111888 > > gcc/ChangeLog: > > * config.gcc: Add AVL propagation PASS. > * config/riscv/riscv-passes.def (INSERT_PASS_AFTER): Ditto. > * config/riscv/riscv-protos.h (make_pass_avlprop): Ditto. > * config/riscv/t-riscv: Ditto. > * config/riscv/riscv-avlprop.cc: New file. > > gcc/testsuite/ChangeLog: > > * gcc.target/riscv/rvv/autovec/partial/select_vl-2.c: Adapt test. > * gcc.target/riscv/rvv/autovec/ternop/ternop_nofm-2.c: Ditto. > * gcc.target/riscv/rvv/autovec/pr111318.c: New test. > * gcc.target/riscv/rvv/autovec/pr111888.c: New test. > > --- > gcc/config.gcc | 2 +- > gcc/config/riscv/riscv-avlprop.cc | 419 ++++++++++++++++++ > gcc/config/riscv/riscv-passes.def | 1 + > gcc/config/riscv/riscv-protos.h | 1 + > gcc/config/riscv/t-riscv | 6 + > .../riscv/rvv/autovec/partial/select_vl-2.c | 5 +- > .../gcc.target/riscv/rvv/autovec/pr111318.c | 16 + > .../gcc.target/riscv/rvv/autovec/pr111888.c | 33 ++ > .../riscv/rvv/autovec/ternop/ternop_nofm-2.c | 1 - > 9 files changed, 480 insertions(+), 4 deletions(-) > create mode 100644 gcc/config/riscv/riscv-avlprop.cc > create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/pr111318.c > create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/pr111888.c > > diff --git a/gcc/config.gcc b/gcc/config.gcc > index 606d3a8513e..efd53965c9a 100644 > --- a/gcc/config.gcc > +++ b/gcc/config.gcc > @@ -544,7 +544,7 @@ pru-*-*) > riscv*) > cpu_type=riscv > extra_objs="riscv-builtins.o riscv-c.o riscv-sr.o > riscv-shorten-memrefs.o riscv-selftests.o riscv-string.o" > - extra_objs="${extra_objs} riscv-v.o riscv-vsetvl.o > riscv-vector-costs.o" > + extra_objs="${extra_objs} riscv-v.o riscv-vsetvl.o > riscv-vector-costs.o riscv-avlprop.o" > extra_objs="${extra_objs} riscv-vector-builtins.o > riscv-vector-builtins-shapes.o riscv-vector-builtins-bases.o" > extra_objs="${extra_objs} thead.o" > d_target_objs="riscv-d.o" > diff --git a/gcc/config/riscv/riscv-avlprop.cc > b/gcc/config/riscv/riscv-avlprop.cc > new file mode 100644 > index 00000000000..2c79ec81806 > --- /dev/null > +++ b/gcc/config/riscv/riscv-avlprop.cc > @@ -0,0 +1,419 @@ > +/* AVL propagation pass for RISC-V 'V' Extension for GNU compiler. > + Copyright (C) 2023-2023 Free Software Foundation, Inc. > + Contributed by Juzhe Zhong (juzhe.zh...@rivai.ai), RiVAI Technologies Ltd. > + > +This file is part of GCC. > + > +GCC is free software; you can redistribute it and/or modify > +it under the terms of the GNU General Public License as published by > +the Free Software Foundation; either version 3, or(at your option) > +any later version. > + > +GCC is distributed in the hope that it will be useful, > +but WITHOUT ANY WARRANTY; without even the implied warranty of > +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +GNU General Public License for more details. > + > +You should have received a copy of the GNU General Public License > +along with GCC; see the file COPYING3. If not see > +<http://www.gnu.org/licenses/>. */ > + > +/* Pre-RA RTL_SSA-based pass propagates AVL for RVV instructions. > + A standalone AVL propagation pass is designed because: > + > + - Better code maintain: > + Current LCM-based VSETVL pass is so complicated that codes > + there will become even harder to maintain. A straight forward > + AVL propagation PASS is much easier to maintain. > + > + - Reduce scalar register pressure: > + A type of AVL propagation is we propagate AVL from NON-VLMAX > + instruction to VLMAX instruction. > + Note: VLMAX instruction should be ignore tail elements (TA) > + and the result should be used by the NON-VLMAX instruction. > + This optimization is mostly for auto-vectorization codes: > + > + vsetvli r136, r137 --- SELECT_VL > + vle8.v (use avl = r136) --- IFN_MASK_LEN_LOAD > + vadd.vv (use VLMAX) --- PLUS_EXPR > + vse8.v (use avl = r136) --- IFN_MASK_LEN_STORE > + > + NO AVL propation: > + > + vsetvli a5, a4, ta > + vle8.v v1 > + vsetvli t0, zero, ta > + vadd.vv v2, v1, v1 > + vse8.v v2 > + > + We can propagate the AVL to 'vadd.vv' since its result > + is consumed by a 'vse8.v' which has AVL = a5 and its > + tail elements are agnostic. > + > + We DON'T do this optimization on VSETVL pass since it is a > + post-RA pass that consumed 't0' already wheras a standalone > + pre-RA AVL propagation pass allows us elide the consumption > + of the pseudo register of 't0' then we can reduce scalar > + register pressure. > + > + - More AVL propagation opportunities: > + A pre-RA pass is more flexible for AVL REG def-use chain, > + thus we will get more potential AVL propagation as long as > + it doesn't increase the scalar register pressure. > +*/ > + > +#define IN_TARGET_CODE 1 > +#define INCLUDE_ALGORITHM > +#define INCLUDE_FUNCTIONAL > + > +#include "config.h" > +#include "system.h" > +#include "coretypes.h" > +#include "tm.h" > +#include "backend.h" > +#include "rtl.h" > +#include "target.h" > +#include "tree-pass.h" > +#include "df.h" > +#include "rtl-ssa.h" > +#include "cfgcleanup.h" > +#include "insn-attr.h" > + > +using namespace rtl_ssa; > +using namespace riscv_vector; > + > +enum avlprop_type > +{ > + /* VLMAX AVL and tail agnostic candidates. */ > + AVLPROP_VLMAX_TA, > + AVLPROP_NONE > +}; > + > +/* dump helper functions */ > +static const char * > +avlprop_type_to_str (enum avlprop_type type) > +{ > + switch (type) > + { > + case AVLPROP_VLMAX_TA: > + return "vlmax_ta"; > + > + default: > + gcc_unreachable (); > + } > +} > + > +static bool > +vlmax_ta_p (rtx_insn *rinsn) > +{ > + return vlmax_avl_type_p (rinsn) && tail_agnostic_p (rinsn); > +} > + > +const pass_data pass_data_avlprop = { > + RTL_PASS, /* type */ > + "avlprop", /* name */ > + OPTGROUP_NONE, /* optinfo_flags */ > + TV_NONE, /* tv_id */ > + 0, /* properties_required */ > + 0, /* properties_provided */ > + 0, /* properties_destroyed */ > + 0, /* todo_flags_start */ > + 0, /* todo_flags_finish */ > +}; > + > +class pass_avlprop : public rtl_opt_pass > +{ > +public: > + pass_avlprop (gcc::context *ctxt) : rtl_opt_pass (pass_data_avlprop, ctxt) > {} > + > + /* opt_pass methods: */ > + virtual bool gate (function *) final override > + { > + return TARGET_VECTOR && optimize > 0; > + } > + virtual unsigned int execute (function *) final override; > + > +private: > + /* The AVL propagation instructions and corresponding preferred AVL. > + It will be updated during the analysis. */ > + hash_map<insn_info *, rtx> *m_avl_propagations; > + > + /* Potential feasible AVL propagation candidates. */ > + auto_vec<std::pair<enum avlprop_type, insn_info *>> m_candidates; > + > + rtx get_preferred_avl (const std::pair<enum avlprop_type, insn_info *>) > const; > + rtx get_vlmax_ta_preferred_avl (insn_info *) const; > + rtx get_nonvlmax_avl (insn_info *) const; > + > + void avlprop_init (function *); > + void avlprop_done (void); > +}; // class pass_avlprop > + > +void > +pass_avlprop::avlprop_init (function *fn) > +{ > + calculate_dominance_info (CDI_DOMINATORS); > + df_analyze (); > + crtl->ssa = new function_info (fn); > + m_avl_propagations = new hash_map<insn_info *, rtx>; > +} > + > +void > +pass_avlprop::avlprop_done (void) > +{ > + free_dominance_info (CDI_DOMINATORS); > + if (crtl->ssa->perform_pending_updates ()) > + cleanup_cfg (0); > + delete crtl->ssa; > + crtl->ssa = nullptr; > + delete m_avl_propagations; > + m_avl_propagations = NULL; > + if (!m_candidates.is_empty ()) > + m_candidates.release (); > +} > + > +/* If we have a preferred AVL to propagate, return the AVL. > + Otherwise, return NULL_RTX as we don't need have any preferred > + AVL. */ > + > +rtx > +pass_avlprop::get_preferred_avl ( > + const std::pair<enum avlprop_type, insn_info *> candidate) const > +{ > + switch (candidate.first) > + { > + case AVLPROP_VLMAX_TA: > + return get_vlmax_ta_preferred_avl (candidate.second); > + default: > + gcc_unreachable (); > + } > + return NULL_RTX; > +} > + > +/* This is a straight forward pattern ALWAYS in paritial auto-vectorization: > + > + VL = SELECT_AVL (AVL, ...) > + V0 = MASK_LEN_LOAD (..., VL) > + V1 = MASK_LEN_LOAD (..., VL) > + V2 = V0 + V1 --- Missed LEN information. > + MASK_LEN_STORE (..., V2, VL) > + > + We prefer PLUS_EXPR (V0 + V1) instead of COND_LEN_ADD (V0, V1, dummy LEN) > + because: > + > + - Few code changes in Loop Vectorizer. > + - Reuse the current clean flow of partial vectorization, That is, apply > + predicate LEN or MASK into LOAD/STORE operations and other special > + arithmetic operations (e.d. DIV), then do the whole vector register > + operation if it DON'T affect the correctness. > + Such flow is used by all other targets like x86, sve, s390, ... etc. > + - PLUS_EXPR has better gimple optimizations than COND_LEN_ADD. > + > + We propagate AVL from NON-VLMAX to VLMAX for gimple IR like PLUS_EXPR > which > + generates the VLMAX instruction due to missed LEN information. The later > + VSETVL PASS will elided the redundant vsetvls. > +*/ > + > +rtx > +pass_avlprop::get_vlmax_ta_preferred_avl (insn_info *insn) const > +{ > + int sew = get_sew (insn->rtl ()); > + enum vlmul_type vlmul = get_vlmul (insn->rtl ()); > + int ratio = calculate_ratio (sew, vlmul); > + > + rtx use_avl = NULL_RTX; > + for (def_info *def : insn->defs ()) > + { > + if (!is_a<set_info *> (def) || def->is_mem ()) > + return NULL_RTX; > + const auto *set = dyn_cast<set_info *> (def); > + > + /* FIXME: Stop AVL propagation if any USE is not a RVV real > + instruction. It should be totally enough for vectorized codes since > + they always locate at extended blocks. > + > + TODO: We can extend PHI checking for intrinsic codes if it > + necessary in the future. */ > + if (!set->is_local_to_ebb ()) > + return NULL_RTX; > + > + for (use_info *use : set->nondebug_insn_uses ()) > + { > + insn_info *use_insn = use->insn (); > + if (!use_insn->can_be_optimized () || use_insn->is_asm () > + || use_insn->is_call () || use_insn->has_volatile_refs () > + || use_insn->has_pre_post_modify () > + || !has_vl_op (use_insn->rtl ()) > + || !tail_agnostic_p (use_insn->rtl ())) > + return NULL_RTX; > + > + int new_sew = get_sew (use_insn->rtl ()); > + enum vlmul_type new_vlmul = get_vlmul (use_insn->rtl ()); > + int new_ratio = calculate_ratio (new_sew, new_vlmul); > + if (new_ratio != ratio) > + return NULL_RTX; > + > + rtx new_use_avl = get_nonvlmax_avl (use_insn); > + if (!new_use_avl || SUBREG_P (new_use_avl)) > + return NULL_RTX; > + if (REG_P (new_use_avl)) > + { > + resource_info resource = full_register (REGNO (new_use_avl)); > + def_lookup dl = crtl->ssa->find_def (resource, use_insn); > + if (dl.matching_set ()) > + return NULL_RTX; > + def_info *def1 = dl.prev_def (insn); > + def_info *def2 = dl.prev_def (use_insn); > + if (!def1 || !def2 || def1 != def2) > + return NULL_RTX; > + > + /* FIXME: We only all AVL propation within a block which should > + be totally enough for vectorized codes. > + > + TODO: We can enhance it here for intrinsic codes in the > future > + if it is necessary. */ > + if (def1->insn ()->bb () != insn->bb () > + && !dominated_by_p (CDI_DOMINATORS, insn->bb ()->cfg_bb (), > + def1->insn ()->bb ()->cfg_bb ())) > + return NULL_RTX; > + if (def1->insn ()->bb () == insn->bb () > + && def1->insn ()->compare_with (insn) >= 0) > + return NULL_RTX; > + } > + > + if (!use_avl) > + use_avl = new_use_avl; > + else if (!rtx_equal_p (use_avl, new_use_avl)) > + return NULL_RTX; > + } > + } > + > + return use_avl; > +} > + > +/* Try to get the NONVLMAX AVL of the INSN. > + INSN can be either NON-VLMAX AVL itself or VLMAX AVL INSN > + before the PASS but has been propagated a NON-VLMAX AVL > + in the before round propagation. */ > +rtx > +pass_avlprop::get_nonvlmax_avl (insn_info *insn) const > +{ > + if (m_avl_propagations->get (insn)) > + return (*m_avl_propagations->get (insn)); > + else if (nonvlmax_avl_type_p (insn->rtl ())) > + { > + extract_insn_cached (insn->rtl ()); > + return recog_data.operand[get_attr_vl_op_idx (insn->rtl ())]; > + } > + > + return NULL_RTX; > +} > + > +/* Main entry point for this pass. */ > +unsigned int > +pass_avlprop::execute (function *fn) > +{ > + avlprop_init (fn); > + > + /* Iterate the whole function in reverse order (which could speed the > + convergence) to collect all potential candidates that could be AVL > + propagated. > + > + Note that: **NOT** all the candidates will be successfully AVL > propagated. > + */ > + for (bb_info *bb : crtl->ssa->reverse_bbs ()) > + { > + for (insn_info *insn : bb->reverse_real_nondebug_insns ()) > + { > + /* We only forward AVL to the instruction that has AVL/VL operand > + and can be optimized in RTL_SSA level. */ > + if (!insn->can_be_optimized () || !has_vl_op (insn->rtl ())) > + continue; > + > + /* TODO: We only do AVL propagation for VLMAX AVL with tail > + agnostic policy since we have missed-LEN information partial > + autovectorization. We could add more more AVL propagation > + for intrinsic codes in the future. */ > + if (vlmax_ta_p (insn->rtl ())) > + m_candidates.safe_push (std::make_pair (AVLPROP_VLMAX_TA, insn)); > + } > + } > + > + if (dump_file && (dump_flags & TDF_DETAILS)) > + { > + fprintf (dump_file, "\nNumber of potential AVL propagations: %d\n", > + m_candidates.length ()); > + for (const auto candidate : m_candidates) > + { > + fprintf (dump_file, "\nAVL propagation type: %s\n", > + avlprop_type_to_str (candidate.first)); > + print_rtl_single (dump_file, candidate.second->rtl ()); > + } > + } > + > + /* Go through all the candidates looking for AVL that we could propagate. > */ > + bool change_p = true; > + while (change_p) > + { > + change_p = false; > + for (auto &candidate : m_candidates) > + { > + rtx new_avl = get_preferred_avl (candidate); > + if (new_avl) > + { > + gcc_assert (!vlmax_avl_p (new_avl)); > + auto &update > + = m_avl_propagations->get_or_insert (candidate.second); > + change_p = !rtx_equal_p (update, new_avl); > + update = new_avl; > + } > + } > + } > + > + if (dump_file && (dump_flags & TDF_DETAILS)) > + fprintf (dump_file, "\nNumber of successful AVL propagations: %d\n\n", > + (int) m_avl_propagations->elements ()); > + > + for (const auto prop : *m_avl_propagations) > + { > + rtx_insn *rinsn = prop.first->rtl (); > + if (dump_file && (dump_flags & TDF_DETAILS)) > + { > + fprintf (dump_file, "\nPropagating AVL: "); > + print_rtl_single (dump_file, prop.second); > + fprintf (dump_file, "into: "); > + print_rtl_single (dump_file, rinsn); > + } > + /* Replace AVL operand. */ > + extract_insn_cached (rinsn); > + rtx avl = recog_data.operand[get_attr_vl_op_idx (rinsn)]; > + int count = count_regno_occurrences (rinsn, REGNO (avl)); > + gcc_assert (count == 1); > + rtx new_pat = simplify_replace_rtx (PATTERN (rinsn), avl, prop.second); > + validate_change_or_fail (rinsn, &PATTERN (rinsn), new_pat, false); > + > + /* Change AVL TYPE into NONVLMAX if it is VLMAX. */ > + if (vlmax_avl_type_p (rinsn)) > + { > + int index = get_attr_avl_type_idx (rinsn); > + gcc_assert (index != INVALID_ATTRIBUTE); > + validate_change_or_fail (rinsn, recog_data.operand_loc[index], > + get_avl_type_rtx (avl_type::NONVLMAX), > + false); > + } > + if (dump_file && (dump_flags & TDF_DETAILS)) > + { > + fprintf (dump_file, "Successfully to match this instruction: "); > + print_rtl_single (dump_file, rinsn); > + } > + } > + > + avlprop_done (); > + return 0; > +} > + > +rtl_opt_pass * > +make_pass_avlprop (gcc::context *ctxt) > +{ > + return new pass_avlprop (ctxt); > +} > diff --git a/gcc/config/riscv/riscv-passes.def > b/gcc/config/riscv/riscv-passes.def > index 4084122cf0a..b6260939d5c 100644 > --- a/gcc/config/riscv/riscv-passes.def > +++ b/gcc/config/riscv/riscv-passes.def > @@ -18,4 +18,5 @@ > <http://www.gnu.org/licenses/>. */ > > INSERT_PASS_AFTER (pass_rtl_store_motion, 1, pass_shorten_memrefs); > +INSERT_PASS_AFTER (pass_split_all_insns, 1, pass_avlprop); > INSERT_PASS_BEFORE (pass_fast_rtl_dce, 1, pass_vsetvl); > diff --git a/gcc/config/riscv/riscv-protos.h b/gcc/config/riscv/riscv-protos.h > index 668d75043ca..d4e17fc3fd0 100644 > --- a/gcc/config/riscv/riscv-protos.h > +++ b/gcc/config/riscv/riscv-protos.h > @@ -156,6 +156,7 @@ extern void riscv_parse_arch_string (const char *, struct > gcc_options *, locatio > extern bool riscv_hard_regno_rename_ok (unsigned, unsigned); > > rtl_opt_pass * make_pass_shorten_memrefs (gcc::context *ctxt); > +rtl_opt_pass * make_pass_avlprop (gcc::context *ctxt); > rtl_opt_pass * make_pass_vsetvl (gcc::context *ctxt); > > /* Routines implemented in riscv-string.c. */ > diff --git a/gcc/config/riscv/t-riscv b/gcc/config/riscv/t-riscv > index dd17056fe82..f8ca3f4ac57 100644 > --- a/gcc/config/riscv/t-riscv > +++ b/gcc/config/riscv/t-riscv > @@ -78,6 +78,12 @@ riscv-vector-costs.o: > $(srcdir)/config/riscv/riscv-vector-costs.cc \ > $(COMPILER) -c $(ALL_COMPILERFLAGS) $(ALL_CPPFLAGS) $(INCLUDES) \ > $(srcdir)/config/riscv/riscv-vector-costs.cc > > +riscv-avlprop.o: $(srcdir)/config/riscv/riscv-avlprop.cc \ > + $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) $(RTL_H) $(REGS_H) \ > + $(TARGET_H) tree-pass.h df.h rtl-ssa.h cfgcleanup.h insn-attr.h > + $(COMPILER) -c $(ALL_COMPILERFLAGS) $(ALL_CPPFLAGS) $(INCLUDES) \ > + $(srcdir)/config/riscv/riscv-avlprop.cc > + > riscv-d.o: $(srcdir)/config/riscv/riscv-d.cc \ > $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) > $(COMPILE) $< > diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/partial/select_vl-2.c > b/gcc/testsuite/gcc.target/riscv/rvv/autovec/partial/select_vl-2.c > index eac7cbc757b..ca88d42cdf4 100644 > --- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/partial/select_vl-2.c > +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/partial/select_vl-2.c > @@ -7,10 +7,11 @@ > /* > ** foo: > ** > vsetivli\t[a-x0-9]+,\s*8,\s*e(8?|16?|32?|64),\s*m(1?|2?|4?|8?|f2?|f4?|f8),\s*t[au],\s*m[au] > +** ... > ** vle32\.v\tv[0-9]+,0\([a-x0-9]+\) > ** ... > -** > vsetvli\t[a-x0-9]+,\s*[a-x0-9]+,\s*e(8?|16?|32?|64),\s*m(1?|2?|4?|8?|f2?|f4?|f8),\s*t[au],\s*m[au] > -** add\t[a-x0-9]+,[a-x0-9]+,[a-x0-9]+ > +** > vsetvli\tzero,\s*[a-x0-9]+,\s*e(8?|16?|32?|64),\s*m(1?|2?|4?|8?|f2?|f4?|f8),\s*t[au],\s*m[au] > +** ... > ** vle32\.v\tv[0-9]+,0\([a-x0-9]+\) > ** ... > */ > diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr111318.c > b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr111318.c > new file mode 100644 > index 00000000000..ff36da8feeb > --- /dev/null > +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr111318.c > @@ -0,0 +1,16 @@ > +/* { dg-do compile } */ > +/* { dg-options "-march=rv64gcv -mabi=lp64d -O3 -fno-vect-cost-model" } */ > + > +void > +foo (int *__restrict a, int *__restrict b, int *__restrict c, int n) > +{ > + for (int i = 0; i < n; i += 1) > + c[i] = a[i] + b[i]; > +} > + > +/* { dg-final { scan-assembler-times {vsetvli} 1 } } */ > +/* { dg-final { scan-assembler-not {vsetivli} } } */ > +/* { dg-final { scan-assembler-times {vsetvli\s*[a-x0-9]+,\s*[a-x0-9]+} 1 } > } */ > +/* { dg-final { scan-assembler-not {vsetvli\s*[a-x0-9]+,\s*zero} } } */ > +/* { dg-final { scan-assembler-not {vsetvli\s*zero} } } */ > +/* { dg-final { scan-assembler-not {vsetivli\s*zero} } } */ > diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr111888.c > b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr111888.c > new file mode 100644 > index 00000000000..2387c20a26c > --- /dev/null > +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr111888.c > @@ -0,0 +1,33 @@ > +/* { dg-do compile } */ > +/* { dg-options "-march=rv64gcv -mabi=lp64d -O3 -fno-vect-cost-model" } */ > + > +void > +foo (int *__restrict a, int *__restrict b, int *__restrict c, > + int *__restrict a2, int *__restrict b2, int *__restrict c2, > + int *__restrict a3, int *__restrict b3, int *__restrict c3, > + int *__restrict a4, int *__restrict b4, int *__restrict c4, > + int *__restrict a5, int *__restrict b5, int *__restrict c5, > + int *__restrict d, int *__restrict d2, int *__restrict d3, > + int *__restrict d4, int *__restrict d5, int n, int m) > +{ > + for (int i = 0; i < n; i++) > + { > + a[i] = b[i] + c[i]; > + a2[i] = b2[i] + c2[i]; > + a3[i] = b3[i] + c3[i]; > + a4[i] = b4[i] + c4[i]; > + a5[i] = a[i] + a4[i]; > + d[i] = a[i] - a2[i]; > + d2[i] = a2[i] * a[i]; > + d3[i] = a3[i] * a2[i]; > + d4[i] = a2[i] * d2[i]; > + d5[i] = a[i] * a2[i] * a3[i] * a4[i] * d[i]; > + } > +} > + > +/* { dg-final { scan-assembler-times {vsetvli} 1 } } */ > +/* { dg-final { scan-assembler-not {vsetivli} } } */ > +/* { dg-final { scan-assembler-times {vsetvli\s*[a-x0-9]+,\s*[a-x0-9]+} 1 } > } */ > +/* { dg-final { scan-assembler-not {vsetvli\s*[a-x0-9]+,\s*zero} } } */ > +/* { dg-final { scan-assembler-not {vsetvli\s*zero} } } */ > +/* { dg-final { scan-assembler-not {vsetivli\s*zero} } } */ > diff --git > a/gcc/testsuite/gcc.target/riscv/rvv/autovec/ternop/ternop_nofm-2.c > b/gcc/testsuite/gcc.target/riscv/rvv/autovec/ternop/ternop_nofm-2.c > index 965365da4bb..13367423751 100644 > --- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/ternop/ternop_nofm-2.c > +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/ternop/ternop_nofm-2.c > @@ -3,7 +3,6 @@ > > #include "ternop-2.c" > > -/* { dg-final { scan-assembler-times {\tvmacc\.vv} 8 } } */ > /* { dg-final { scan-assembler-times {\tvfma[c-d][c-d]\.vv} 9 } } */ > /* { dg-final { scan-tree-dump-times "COND_LEN_FMA" 9 "optimized" } } */ > /* { dg-final { scan-assembler-not {\tvmv} } } */ > -- > 2.36.3 >