https://gcc.gnu.org/bugzilla/show_bug.cgi?id=126385
ptomsich at gcc dot gnu.org changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |ptomsich at gcc dot gnu.org
--- Comment #2 from ptomsich at gcc dot gnu.org ---
Yes, N separate reduction vectorizations is what we want.
[Summarizing from internal ticket with the failed attempt to get this into the
loop-vectorizer...]
The single-vector-IV form has no emittable shape regardless of the gather: the
integer dot-product ISAs (sdot/udot/usdot, VNNI) accumulate within a register
only. There is no per-lane dot product with per-lane operands.
The N streams must be N physical accumulators, each fed by a unit-stride row
stream. Unroll-and-jam produces exactly that, and dissolves the gather: each
copy's weight access becomes consecutive [j] indexing off its own base.
After the jam the loop vectorizer needs no changes: one inner loop, N
independent reduction PHIs, N DOT_PROD chains. If outer-loop vectorization
grows a multi-stream reduction mode, the jam gating can retire in its favour.
We tried to extend the outer-loop vectorizer for a multi-stream reduction, but
ran into roadblocks (from the loop vectorizer's design/assumptions) on multiple
levels:
1. code generation: the outer-loop path executes the inner loop sequentially
once per outer iteration (see the comment in vect_transform_loop); N concurrent
inner-loop instances would need a replicated-body strategy, i.e. a new
code-generation path rather than a parameter of the existing one.
2. representation: the vectorizer maps one scalar statement to one vector
statement with lanes drawn from a single loop dimension, but the N streams are
not lanes of anything (each needs its own scalar base, its own unit-stride
data-ref and its own reduction PHI ...). We tried relaxing the grouped-access
rejection: recognition proceeds, ~24 vect-outer-* tests break, and the
nested-cycle MAC still has no reduction def_type, so vectorizable_reduction
refuses).
3. discovery: SLP actively works in the wrong direction here, collecting
independent reductions into one reduc_group with a single vector IV (for
multi-stream it would have to be prevented, not extended).
We eventually gave up, as our assessment was that multi-stream outer-loop
vectorization would amounts to a second, replicated-inner-loop vectorizer.
So we settled on unroll-and-jam as the existing machinery already handles our
case.
Finally, the jammed form is profitable even without vectorization. With
-fno-tree-vectorize the pass leaves 4 scalar MAC chains; llvm-mca (neoverse-n1
model, 100 iterations) on the two inner-loop bodies:
unjammed (1 output/iter): ldrsb+ldrb+madd chain, 308 cycles
-> 3.08 cycles per output element
jammed (4 outputs/iter): 1 shared ldrb + 4x (ldrsb+madd),
707 cycles -> 1.77 cycles per output
element, 1.74x
The single chain is bound by the loop-carried madd latency (3 cycles on N1).
Four independent accumulators fill those stall cycles, and the input load is
fetched once per iteration instead of once per stream. So the transform pays
off from instruction-level parallelism alone without DOT_PROD.
