[Bug tree-optimization/128140] New: [17 Regression] wrong code at -O2: vectorized x >= C ? x : C - 1 gives a wrong result in the lane where C wraps around to 0

[yuki.tanaka at defaultcase dot de via Gcc-bugs]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=128140

            Bug ID: 128140
           Summary: [17 Regression] wrong code at -O2: vectorized x >= C ?
                    x : C - 1 gives a wrong result in the lane where C
                    wraps around to 0
           Product: gcc
           Version: 17.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: yuki.tanaka at defaultcase dot de
  Target Milestone: ---

unsigned char x[8], r[8];

__attribute__((noipa)) void a(void)
{
  for (int i = 0; i < 8; i++)
    r[i] = x[i] >= (unsigned char) (i + 253) ? x[i] : (unsigned char) (i +
252);
}

int main(void)
{
  a();
  if (r[3] != 0)
    __builtin_abort();
  return 0;
}

Compiler Explorer (GCC 16.1 vs. 17):
https://godbolt.org/z/hMvhh6Wsh

GCC 17:
"a":
        movq    xmm1, QWORD PTR "x"[rip]
        movabs  rax, 216736835873734140
        movq    xmm0, rax
        pmaxub  xmm0, xmm1
        movq    QWORD PTR "r"[rip], xmm0
        ret

The loop is turned into a single MAX with i + 252.  That is only correct if i +
252 == (i + 253) - 1 in every lane, but in lane 3 i + 253 wraps to 0 and i +
252 is 255, so r[3] is 255 instead of 0. GCC 16.1 keeps the compare and select.
 Also fails at -O3, on aarch64 -O2 trunk emits the same single umax.

Possibly related to r17-3277-g729eb68a002 ("tree-optimization: Fold
canonicalized vector min/max conditionals [PR98602]").