felipepessoto commented on PR #12388:
URL: https://github.com/apache/gluten/pull/12388#issuecomment-4928578615
Heads-up for reviewers: since the last review round I added **flaky-test
handling** to the gate (it wasn't in the original description), and just
reworked it to match by **error signature** rather than test name. Summarising
so nobody has to reverse-engineer it from the diffs.
**The problem.** A handful of Delta MERGE-with-deletion-vector tests fail
*non-deterministically*: the same bundle passes them on one run and fails them
on the next. They can't go in `known-failures.txt` — baselining them reds the
gate when they pass (`fail_on_fixed`), and leaving them out reds it when they
fail. The root cause is a native bug: the DV bitmap aggregator aborts on an
invalid `Long.MAX_VALUE` row index during a DV-writing MERGE
(`RoaringBitmapArray.cpp` `addSafe`, `value <= kMaxRepresentableValue`). It's
intermittent and lands on a **different `*DVs*Suite` MERGE test each run**.
**What the gate now does.** Two ways to quarantine a flake (a quarantined
failure is neutral — never a regression, never now-passing, and excluded from
the regenerated baseline):
1. `flaky-tests.txt` — by **test name** (`<suite-glob>#<test>`), for
ordinary flakes.
2. `flaky-error-patterns.txt` — by **error signature**: a regex matched
against the failed test's JUnit `<failure>`/`<error>` text. Seeded with the
DV-bitmap signature:
```
Delta RoaringBitmapArray row index \d+ exceeds max representable value
```
Because this intermittent bug hits a different test each run, matching by
name was whack-a-mole (it had grown to 6 entries). Matching the signature
ignores it on **whichever** test it lands, so I removed those 6 name entries.
It's also *more precise* than a name glob: a **different** real failure in the
same DV suite is still caught as a regression, since only failures carrying the
signature are ignored.
This is an **interim** measure — it hides a real native bug from CI. The
pattern references the tracking issue and should be removed once the row-index
bug is fixed in the native backend.
Verified against a real failing shard report: the DV failure is reported as
a regression without the patterns file, and quarantined (gate green) with it.
See `.github/workflows/util/delta-spark-ut/README.md` → "Flaky tests" for
details. Relevant commits: `0a3759f77` (initial flaky quarantine) and
`cd024df86` (signature-based).
I've refreshed the PR description to cover this (plus the Arrow-jar removal
and the script extraction below).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
