On 09/06/2026 10:24, Richard Biener wrote:
On Wed, Jun 3, 2026 at 5:20 PM Christopher Bazley <[email protected]> wrote:Add two new fields to SLP tree nodes, which are accessed as SLP_TREE_CAN_USE_PARTIAL_VECTORS_P and SLP_TREE_PARTIAL_VECTORS_STYLE. SLP_TREE_CAN_USE_PARTIAL_VECTORS_P is analogous to the existing predicate LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P. It is initialized to true. This flag just records whether the target could vectorize a node using a partial vector; it does not say anything about whether the vector actually is partial, or how the target would support use of a partial vector. Some kinds of node require mask/length for partial vectors; others don't. In the latter case (e.g., for add operations), SLP_TREE_CAN_USE_PARTIAL_VECTORS_P will remain true. SLP_TREE_PARTIAL_VECTORS_STYLE is analogous to the existing field LOOP_VINFO_PARTIAL_VECTORS_STYLE. Both are initialized to 'none'. The vect_partial_vectors_avx512 enumerator is not used for BB SLP. Unlike loop vectorization, a different style of partial vectors can be chosen for each node during analysis of that node. Implement the recently-introduced wrapper functions, vect_record_(len|mask), for BB SLP by setting SLP_TREE_PARTIAL_VECTORS_STYLE to indicate that a mask or length should be used for a given SLP node. The passed-in vec_info is ignored. Implement the vect_fully_(masked|with_length)_p wrapper functions for BB SLP by checking the SLP_TREE_PARTIAL_VECTORS_STYLE. This should be sufficient because at most one of vect_record_(len|mask) and vect_cannot_use_partial_vectors are expected to be called for any given SLP node. SLP_TREE_CAN_USE_PARTIAL_VECTORS_P should be true if the style is not 'none', but its value isn't used beyond the analysis phase. The implementations of vect_get_mask and vect_get_len for BB SLP are non-trivial (albeit simpler than for loop vectorization), therefore they are delegated to SLP-specific functions defined in tree-vect-slp.cc. Implement the vect_cannot_use_partial_vectors wrapper function by setting the SLP_TREE_CAN_USE_PARTIAL_VECTORS_P flag to false. To prevent regressions, vect_can_use_partial_vectors_p still returns false for BB SLP regardless (for now). This prevents vect_record_mask or vect_record_len from being called. gcc/ChangeLog: * tree-vect-slp.cc (_slp_tree::_slp_tree): initialize new partial_vector_style, can_use_partial_vectors and num_partial_vectors members. (vect_slp_analyze_node_operations): Account for worst-case prologue costs of per-node partial-vector mask or length materialisation. (vect_slp_record_bb_style): Set the partial vector style of an SLP node, checking that the style does not flip-flop between mask and length. (vect_slp_record_bb_mask): Use vect_slp_record_bb_style to set the partial vector style of the SLP tree node to vect_partial_vectors_while_ult. (vect_slp_get_bb_mask): New function to materialize a mask for basic block SLP vectorization. (vect_slp_record_bb_len): Use vect_slp_record_bb_style to set the partial vector style of the SLP tree node to vect_partial_vectors_len. (vect_slp_get_bb_len): New function to materialize a length for basic block SLP vectorization. * tree-vect-stmts.cc (vectorizable_internal_function): (vect_record_mask): Handle the basic block SLP use case by delegating to vect_slp_record_bb_mask. (vect_get_mask): Handle the basic block SLP use case by delegating to vect_slp_get_bb_mask. (vect_record_len): Handle the basic block SLP use case by delegating to vect_slp_record_bb_len. (vect_get_len): Handle the basic block SLP use case by delegating to vect_slp_get_bb_len. (vect_gen_while_ssa_name): New function containing code refactored out of vect_gen_while for reuse by vect_slp_get_bb_mask. (vect_gen_while): Use vect_gen_while_ssa_name instead of custom code for some of the implementation. * tree-vectorizer.h (enum vect_partial_vector_style): Move this definition earlier to allow reuse by struct _slp_tree. (struct _slp_tree): Add a partial_vector_style member to record whether to use a length or mask for the SLP tree node, if partial vectors are required and supported. Add a can_use_partial_vectors member to record whether partial vectors are supported for the SLP tree node. Add a num_partial_vectors member for costing. (SLP_TREE_PARTIAL_VECTORS_STYLE): New member accessor macro. (SLP_TREE_CAN_USE_PARTIAL_VECTORS_P): New member accessor macro. (SLP_TREE_NUM_PARTIAL_VECTORS): New member accessor macro. (vect_gen_while_ssa_name): Declaration of a new function. (vect_slp_get_bb_mask): As above. (vect_slp_get_bb_len): As above. (vect_cannot_use_partial_vectors): Handle the basic block SLP use-case by setting SLP_TREE_CAN_USE_PARTIAL_VECTORS_P to false. (vect_fully_with_length_p): Handle the basic block SLP use case by checking whether the SLP_TREE_PARTIAL_VECTORS_STYLE is vect_partial_vectors_len. (vect_fully_masked_p): Handle the basic block SLP use case by checking whether the SLP_TREE_PARTIAL_VECTORS_STYLE is vect_partial_vectors_while_ult. --- gcc/tree-vect-slp.cc | 182 +++++++++++++++++++++++++++++++++++++++++ gcc/tree-vect-stmts.cc | 52 +++++++----- gcc/tree-vectorizer.h | 52 ++++++++---- 3 files changed, 247 insertions(+), 39 deletions(-) diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc index 075e93f04a9..4dd7e6e1e21 100644 --- a/gcc/tree-vect-slp.cc +++ b/gcc/tree-vect-slp.cc @@ -125,6 +125,9 @@ _slp_tree::_slp_tree () SLP_TREE_GS_BASE (this) = NULL_TREE; this->ldst_lanes = false; this->avoid_stlf_fail = false; + SLP_TREE_PARTIAL_VECTORS_STYLE (this) = vect_partial_vectors_none; + SLP_TREE_CAN_USE_PARTIAL_VECTORS_P (this) = true; + SLP_TREE_NUM_PARTIAL_VECTORS (this) = 0; SLP_TREE_VECTYPE (this) = NULL_TREE; SLP_TREE_REPRESENTATIVE (this) = NULL; this->cycle_info.id = -1; @@ -8958,6 +8961,40 @@ vect_slp_analyze_node_operations (vec_info *vinfo, slp_tree node, vect_prologue_cost_for_slp (vinfo, child, cost_vec); } + if (res) + { + /* Take care of special costs for partial vectors. + Costing each partial vector is excessive for many SLP instances, + because it is common to materialise identical masks/lengths for related + operations (e.g., for vector loads and stores of the same length). + Masks/lengths can also be shared between SLP subgraphs or eliminated by + pattern-based lowering during instruction selection. However, it's + simpler and safer to use the worst-case cost; if this ends up being the + tie-breaker between vectorizing or not, then it's probably better not + to vectorize. */I'd prefer to do this per SLP subgraph group based on recorded requirements so similar how loop masking is set up.
Thank you for providing clear direction. I have implemented that in what I hope will soon become version 13 of my patchset.
+ const int num_partial_vectors = SLP_TREE_NUM_PARTIAL_VECTORS (node); + + if (SLP_TREE_PARTIAL_VECTORS_STYLE (node) + == vect_partial_vectors_while_ult) + { + gcc_assert (num_partial_vectors > 0); + record_stmt_cost (cost_vec, num_partial_vectors, vector_stmt, NULL, + NULL, NULL_TREE, 0, vect_prologue); + } + else if (SLP_TREE_PARTIAL_VECTORS_STYLE (node) + == vect_partial_vectors_len) + { + /* Need to set up a length in the prologue. */ + gcc_assert (num_partial_vectors > 0); + record_stmt_cost (cost_vec, num_partial_vectors, scalar_stmt, NULL, + NULL, NULL_TREE, 0, vect_prologue); + } + else + { + gcc_assert (num_partial_vectors == 0); + } + } + /* If this node or any of its children can't be vectorized, try pruning the tree here rather than felling the whole thing. */ if (!res && vect_slp_convert_to_external (vinfo, node, node_instance)) @@ -12441,3 +12478,148 @@ vect_schedule_slp (vec_info *vinfo, const vec<slp_instance> &slp_instances) } } } + +/* Record that a specific partial vector style could be used to vectorize + SLP_NODE if required. */ + +static void +vect_slp_record_bb_style (slp_tree slp_node, vect_partial_vector_style style) +{ + gcc_assert (style != vect_partial_vectors_none); + gcc_assert (style != vect_partial_vectors_avx512); + + if (SLP_TREE_PARTIAL_VECTORS_STYLE (slp_node) == vect_partial_vectors_none) + SLP_TREE_PARTIAL_VECTORS_STYLE (slp_node) = style; + else + gcc_assert (SLP_TREE_PARTIAL_VECTORS_STYLE (slp_node) == style); +} + +/* Record that a complete set of masks associated with SLP_NODE would need to + contain a sequence of NVECTORS masks that each control a vector of type + VECTYPE. If SCALAR_MASK is nonnull, the fully-masked loop would AND + these vector masks with the vector version of SCALAR_MASK. */ +void +vect_slp_record_bb_mask (slp_tree slp_node, unsigned int /* nvectors */, + tree /* vectype */, tree /* scalar_mask */) +{ + vect_slp_record_bb_style (slp_node, vect_partial_vectors_while_ult); + + /* FORNOW: this often overestimates the number of masks for costing purposes + because, after lowering, masks have often been eliminated, shared between + SLP nodes, or even shared between SLP subgraphs. */ + SLP_TREE_NUM_PARTIAL_VECTORS(slp_node) ++; +} + +/* Materialize mask number INDEX for a group of scalar stmts in SLP_NODE that + operate on NVECTORS vectors of type VECTYPE, where 0 <= INDEX < NVECTORS. + Insert any set-up statements before GSI. */ + +tree +vect_slp_get_bb_mask (slp_tree slp_node, gimple_stmt_iterator *gsi, + unsigned int nvectors, tree vectype, unsigned int index) +{ + gcc_assert (SLP_TREE_PARTIAL_VECTORS_STYLE (slp_node) + == vect_partial_vectors_while_ult); + gcc_assert (nvectors >= 1); + gcc_assert (index < nvectors); + + const poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype); + const unsigned int group_size = SLP_TREE_LANES (slp_node); + unsigned int mask_size = group_size; + const tree masktype = truth_type_for (vectype); + + if (nunits.is_constant ()) + { + /* Only the last vector can be a partial vector. */ + if (index + 1 < nvectors) + return build_minus_one_cst (masktype); + + /* Return a mask for a possibly-partial tail vector. */ + const unsigned int const_nunits = nunits.to_constant (); + const unsigned int head_size = (nvectors - 1) * const_nunits; + gcc_assert (head_size <= group_size); + mask_size = group_size - head_size; + + if (mask_size == const_nunits) + return build_minus_one_cst (masktype); + } + else + { + /* Return a mask for a single variable-length vector. */ + gcc_assert (nvectors == 1); + gcc_assert (known_le (mask_size, nunits)); + } + + /* FORNOW: don't bother maintaining a set of mask constants to allow + sharing between nodes belonging to the same instance of bb_vec_info + or even within the same SLP subgraph. */See above. The loop code already should have everything set up for caching. Why not reuse that?
When I was creating the RFC and then debugging various issues exposed by use of partial vectors with BB SLP vectorisation, I wanted my changes to be as minimal as possible.
I have since implemented caching. It appears not to provide any observable sharing in the AArch64 assembly language produced for WHILE_ULT-style partial vectors because the 'fre' pass replaces the resultant uses of VIEW_CONVERT_EXPR with separate constants (even in cases where two extra instructions are required).
Reusing the same caching mechanism as the loop vectoriser proved a bit tricky, because whereas WHILE_ULT-style caching of partial vector requirements for a loop can use the number of vectors as the top-level cache index, the number of vectors is always one for BB SLP with a predicated tail. Instead of recording a tuple of vector type and number of vectors, vect_record_mask must therefore record a tuple of vector type and SLP group size.
The array of rgroup controls must likewise be indexed by a value derived from the SLP group size instead of by the number of vectors. This new SLP-specific index cannot actually be the group size though, because reusability of masks built using WHILE_ULT does not depend on group size alone, but compatible combinations of group size and vector type.
For example these two mask requirements must share an rgroup: vectype = VNx16QI nscalars_per_iter = SLP_TREE_LANES = 12 key = 12 * max(16,4) / 16 = 12 vectype = VNx4SI nscalars_per_iter = SLP_TREE_LANES = 3 key = 3 * max(16,4) / 4 = 12 max_nscalars_per_iter = 12 type = truth_type_for (VNx16QI) factor = 1Another new scheme is also needed for caching AVX512-style masks, in which the cache index actually is the SLP group_size.
I am not yet confident about the support for LEN-style partial vectors that I have reimplemented because this constraint looks tricky to satisfy:
/* For now, we only support cases in which all loads and stores fall back
to VnQI or none do. */
gcc_assert (!rgl->max_nscalars_per_iter
|| (rgl->factor == 1 && factor == 1)
|| (rgl->max_nscalars_per_iter * rgl->factor
== nscalars_per_iter * factor));
I don't know how common calls to vect_record_len with factor != 1 might
be, given that they happen when vecmode != vmode:
machine_mode mask_mode, vmode;
if (get_len_load_store_mode (vecmode, is_load, nullptr, elsvals)
.exists (&vmode))
+ gimple_seq stmts = NULL; + const tree cmp_type = size_type_node; + const tree start_index = build_zero_cst (cmp_type); + const tree end_index = build_int_cst (cmp_type, mask_size); + const tree mask = make_temp_ssa_name (masktype, NULL, "slp_mask"); + vect_gen_while_ssa_name (&stmts, masktype, start_index, end_index, mask); + gsi_insert_seq_before (gsi, stmts, GSI_SAME_STMT); + return mask; +} + +/* Record that a complete set of lengths associated with SLP_NODE would need to + contain a sequence of NVECTORS lengths for controlling an operation on + VECTYPE. The operation splits each element of VECTYPE into FACTOR separate + subelements, measuring the length as a number of these subelements. */ + +void +vect_slp_record_bb_len (slp_tree slp_node, unsigned int /* nvectors */, + tree /* vectype */, unsigned int /* factor */) +{ + vect_slp_record_bb_style (slp_node, vect_partial_vectors_len); + + /* FORNOW: this probably overestimates the number of lengths for costing + purposes because, after lowering, lengths might have been eliminated, + shared between SLP nodes, or even shared between SLP subgraphs. */ + SLP_TREE_NUM_PARTIAL_VECTORS (slp_node)++; +} + +/* Materialize length number INDEX for a group of scalar stmts in SLP_NODE that + operate on NVECTORS vectors of type VECTYPE, where 0 <= INDEX < NVECTORS. + Return a value that contains FACTOR multiplied by the number of elements that + should be processed. */ + +tree +vect_slp_get_bb_len (slp_tree slp_node, unsigned int nvectors, tree vectype, + unsigned int index, unsigned int factor, bool adjusted) +{ + gcc_checking_assert (SLP_TREE_PARTIAL_VECTORS_STYLE (slp_node) + == vect_partial_vectors_len); + gcc_assert (nvectors >= 1); + gcc_assert (index < nvectors); + (void) adjusted; + + const poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype); + const unsigned int group_size = SLP_TREE_LANES (slp_node); + unsigned int len = group_size; + + if (nunits.is_constant ()) + { + const unsigned int const_nunits = nunits.to_constant (); + + /* Only the last vector can be a partial vector. */ + if (index + 1 < nvectors) + len = const_nunits; + else + { + /* Return a length for a possibly-partial tail vector. */ + const unsigned int head_size = (nvectors - 1) * const_nunits; + gcc_assert (head_size <= group_size); + len = group_size - head_size; + } + } + else + { + /* Return a length for a single variable-length vector. */ + gcc_assert (nvectors == 1); + gcc_assert (known_le (len, nunits)); + } + + return size_int (len * factor); +} diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index 15fca17a407..ecad74e7cbf 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -1385,7 +1385,9 @@ vectorizable_internal_function (combined_fn cfn, tree fndecl, /* Record that a complete set of masks associated with VINFO would need to contain a sequence of NVECTORS masks that each control a vector of type VECTYPE. If SCALAR_MASK is nonnull, the fully-masked loop would AND - these vector masks with the vector version of SCALAR_MASK. */ + these vector masks with the vector version of SCALAR_MASK. Alternatively, + if doing basic block vectorization, record that a mask could be used to + vectorize SLP_NODE if required. */ static void vect_record_mask (vec_info *vinfo, slp_tree slp_node, unsigned int nvectors, tree vectype, tree scalar_mask) @@ -1395,7 +1397,7 @@ vect_record_mask (vec_info *vinfo, slp_tree slp_node, unsigned int nvectors, vect_record_loop_mask (loop_vinfo, &LOOP_VINFO_MASKS (loop_vinfo), nvectors, vectype, scalar_mask); else - (void) slp_node; /* FORNOW */ + vect_slp_record_bb_mask (slp_node, nvectors, vectype, scalar_mask); } /* Given a complete set of masks associated with VINFO, extract mask number @@ -1413,16 +1415,15 @@ vect_get_mask (vec_info *vinfo, slp_tree slp_node, gimple_stmt_iterator *gsi, return vect_get_loop_mask (loop_vinfo, gsi, &LOOP_VINFO_MASKS (loop_vinfo), nvectors, vectype, index); else - { - (void) slp_node; /* FORNOW */ - return NULL_TREE; - } + return vect_slp_get_bb_mask (slp_node, gsi, nvectors, vectype, index); } /* Record that a complete set of lengths associated with VINFO would need to contain a sequence of NVECTORS lengths for controlling an operation on VECTYPE. The operation splits each element of VECTYPE into FACTOR separate - subelements, measuring the length as a number of these subelements. */ + subelements, measuring the length as a number of these subelements. + Alternatively, if doing basic block vectorization, record that a length limit + could be used to vectorize SLP_NODE if required. */ static void vect_record_len (vec_info *vinfo, slp_tree slp_node, unsigned int nvectors, tree vectype, unsigned int factor) @@ -1432,7 +1433,7 @@ vect_record_len (vec_info *vinfo, slp_tree slp_node, unsigned int nvectors, vect_record_loop_len (loop_vinfo, &LOOP_VINFO_LENS (loop_vinfo), nvectors, vectype, factor); else - (void) slp_node; /* FORNOW */ + vect_slp_record_bb_len (slp_node, nvectors, vectype, factor); } /* Given a complete set of lengths associated with VINFO, extract length number @@ -1453,10 +1454,8 @@ vect_get_len (vec_info *vinfo, slp_tree slp_node, gimple_stmt_iterator *gsi, return vect_get_loop_len (loop_vinfo, gsi, &LOOP_VINFO_LENS (loop_vinfo), nvectors, vectype, index, factor, adjusted); else - { - (void) slp_node; /* FORNOW */ - return NULL_TREE; - } + return vect_slp_get_bb_len (slp_node, nvectors, vectype, index, factor, + adjusted); } static tree permute_vec_elements (vec_info *, tree, tree, tree, stmt_vec_info, @@ -14710,24 +14709,35 @@ supportable_indirect_convert_operation (code_helper code, mask[I] is true iff J + START_INDEX < END_INDEX for all J <= I. Add the statements to SEQ. */ +void +vect_gen_while_ssa_name (gimple_seq *seq, tree mask_type, tree start_index, + tree end_index, tree ssa_name) +{ + tree cmp_type = TREE_TYPE (start_index); + gcc_checking_assert (direct_internal_fn_supported_p (IFN_WHILE_ULT, cmp_type, + mask_type, + OPTIMIZE_FOR_SPEED)); + gcall *call + = gimple_build_call_internal (IFN_WHILE_ULT, 3, start_index, end_index, + build_zero_cst (mask_type)); + gimple_call_set_lhs (call, ssa_name); + gimple_seq_add_stmt (seq, call); +} + +/* Like vect_gen_while_ssa_name except that it creates a new SSA_NAME node + for type MASK_TYPE defined in the created GIMPLE_CALL statement. If NAME + is not a null pointer then it is used for the SSA_NAME in dumps. */ + tree vect_gen_while (gimple_seq *seq, tree mask_type, tree start_index, tree end_index, const char *name) { - tree cmp_type = TREE_TYPE (start_index); - gcc_checking_assert (direct_internal_fn_supported_p (IFN_WHILE_ULT, - cmp_type, mask_type, - OPTIMIZE_FOR_SPEED)); - gcall *call = gimple_build_call_internal (IFN_WHILE_ULT, 3, - start_index, end_index, - build_zero_cst (mask_type)); tree tmp; if (name) tmp = make_temp_ssa_name (mask_type, NULL, name); else tmp = make_ssa_name (mask_type); - gimple_call_set_lhs (call, tmp); - gimple_seq_add_stmt (seq, call); + vect_gen_while_ssa_name (seq, mask_type, start_index, end_index, tmp); return tmp; } diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index a3855568b09..f79f04ff8ac 100644 --- a/gcc./tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -312,6 +312,13 @@ struct vect_load_store_data : vect_data { bool subchain_p; // VMAT_STRIDED_SLP and VMAT_GATHER_SCATTER }; +enum vect_partial_vector_style { + vect_partial_vectors_none, + vect_partial_vectors_while_ult, + vect_partial_vectors_avx512, + vect_partial_vectors_len +}; + /* A computation tree of an SLP instance. Each node corresponds to a group of stmts to be packed in a SIMD stmt. */ struct _slp_tree { @@ -377,7 +384,16 @@ struct _slp_tree { /* For BB vect, flag to indicate this load node should be vectorized as to avoid STLF fails because of related stores. */ bool avoid_stlf_fail; - + /* The style used for implementing partial vectors if LANES is less than + the minimum number of lanes implied by the VECTYPE. */ + vect_partial_vector_style partial_vector_style;I wonder if we want to / need to mix style across the SLP subgraph, likewise
Doesn't partial vector style simply reflect target-specific capabilities that should be constant for the whole SLP graph?
whether we really need to track can_use_partial_vectors per SLP node as opposed to per subgraph. Likewise I wonder if we want to deal with the
I invented the vect_can_use_partial_vectors_p and vect_cannot_use_partial_vectors functions to avoid having to change existing flow control in the vectorizable_* functions. Since those functions now return false if partial vectors are required but not available (instead of merely tracking whether partial vectors can be used and deferring checking of whether they are actually required until later), it no longer seems meaningful to track can_use_partial_vectors per SLP node.
case of parts of the graph being unsupported because of lack of masking support which we could fix by promoting that part extern (not covered) rather than failing the whole subgraph. That is, I'm questioning (maybe again?) the overall tracking/analysis phase?
It wasn't quite right anyway because vect_slp_record_bb_mask could set style vect_partial_vectors_while_ult for an SLP node without actually checking the result of direct_internal_fn_supported_p (IFN_WHILE_ULT, ...). That is the bug reported by Hongtao.
With the revised design in what will become version 13 of my patchset, the analysis phase (still) gives up trying to vectorise an SLP node if internal functions required to consume masks are unavailable. It no longer neglects to check the result of direct_internal_fn_supported_p (IFN_WHILE_ULT, ...) but does not do so until vect_verify_full_masking is called after analyzing all statements in the SLP region.
This is closer to the design of the loop vectoriser, but seems to preclude doing anything to fix up parts of the graph for which masks cannot be produced.
Thanks, -- Christopher Bazley Staff Software Engineer, GNU Tools Team. Arm Ltd, 110 Fulbourn Road, Cambridge, CB1 9NJ, UK. http://www.arm.com/
