https://gcc.gnu.org/bugzilla/show_bug.cgi?id=126385
Bug ID: 126385
Summary: aarch64: integer GEMV reduction vectorized as one
serial dot-product chain instead of parallel
accumulators
Product: gcc
Version: 17.0
Status: UNCONFIRMED
Keywords: missed-optimization
Severity: normal
Priority: P3
Component: tree-optimization
Assignee: unassigned at gcc dot gnu.org
Reporter: konstantinos.eleftheriou at vrull dot eu
Target Milestone: ---
Target: aarch64-*-*
Created attachment 65122
--> https://gcc.gnu.org/bugzilla/attachment.cgi?id=65122&action=edit
reduced_testcase
An integer matrix-vector multiply (the affine / GEMV dot product,
sum += weights[i][j] * input[j]) gets vectorized on the inner loop into a
single usdot accumulator. That accumulator is loop-carried, so the inner
loop runs at the usdot latency, one dot product deep, even though the
outer loop hands us plenty of independent output rows to work on at the
same time.
Compiling the testcase with `-O2 -mcpu=ampere1a -S reduced_testcase.c`
generates following for the inner loop:
.L2:
ldr q30, [x1, x0]
ldr q29, [x3, x0]
add x0, x0, 16
usdot v31.4s, v29.16b, v30.16b
cmp x0, 512
bne .L2
Every usdot reads and writes v31, so each iteration waits on the one
before it.
Since the output rows are independent, the way out is to run several
accumulators at once. Writing the same kernel to compute four output
rows per outer iteration, with four accumulators and one shared input
load, gets GCC to emit exactly that:
for (int i = 0; i < OUT_DIM; i += 4)
{
int32_t s0 = biases[i+0], s1 = biases[i+1];
int32_t s2 = biases[i+2], s3 = biases[i+3];
const int8_t *w0 = weights + (i+0) * IN_DIM;
const int8_t *w1 = weights + (i+1) * IN_DIM;
const int8_t *w2 = weights + (i+2) * IN_DIM;
const int8_t *w3 = weights + (i+3) * IN_DIM;
for (int j = 0; j < IN_DIM; ++j)
{
s0 += (int32_t) w0[j] * (int32_t) input[j];
s1 += (int32_t) w1[j] * (int32_t) input[j];
s2 += (int32_t) w2[j] * (int32_t) input[j];
s3 += (int32_t) w3[j] * (int32_t) input[j];
}
out[i+0] = s0; out[i+1] = s1;
out[i+2] = s2; out[i+3] = s3;
}
.L2:
ldr q31, [x3, x1]
ldr q26, [x5, x1]
usdot v30.4s, v31.16b, v26.16b
ldr q26, [x6, x1]
usdot v29.4s, v31.16b, v26.16b
ldr q26, [x7, x1]
usdot v27.4s, v31.16b, v26.16b
ldr q26, [x4, x1]
add x1, x1, 16
usdot v28.4s, v31.16b, v26.16b
cmp x1, 512
bne .L2
Four independent accumulators (v30, v29, v27, v28) and one shared input
load, so the loop is no longer latency-bound.
So GCC can generate the parallel form, it just will not derive it from
the natural single-output source. Everything it needs is there:
independent output rows in the outer loop and a reduction over the inner
loop.
This shows up in SPEC2026's 706.stockfish_r benchmark.
Suggested fix:
Extend unroll-and-jam: strip-mine the outer loop by N and fuse
the N inner reductions into one body that keeps N independent
accumulator PHIs, rather than collapsing them into a single serial chain
the way plain jamming would. The vectorizer then sees one inner loop with N
independent reductions and emits N parallel dot-product accumulators.
