felipepessoto opened a new issue, #12743:
URL: https://github.com/apache/gluten/issues/12743
The Delta Spark UT pipeline landed in #12388 (fixes #9296). It runs
delta-io/delta's `spark`
test suite against a Gluten Velox bundle and gates each run against a
committed baseline of
known failures.
That baseline is currently **735 known failures out of 19,073 tests**. This
issue tracks the
work to bring that number down and to improve the pipeline itself. It is a
tracking issue only —
discussion belongs in the linked issues.
## 1. Correctness / offload gaps behind the 735 baseline entries
The largest clusters, counted from `known-failures.txt` by suite:
| Tests | Area | Issue |
|---:|---|---|
| 130 | `DescribeDeltaHistorySuite` + `...WithCatalogOwnedBatch100Suite` |
needs triage |
| 100 | `stats.DataSkippingDeltaV1*` (7 suite variants, 34 of them
column-mapping) | needs triage |
| 64 | `ImplicitMergeCastingSuite` + `ImplicitStreamingMergeCastingSuite`
(overflow / storeAssignmentPolicy) | needs triage |
| 42 | `commands.backfill.RowTrackingBackfillConflicts(DV)Suite` | needs
triage |
| 14 | `io.delta.tables.DeltaTableHadoopOptionsSuite` — Hadoop FS options |
#4963 |
Two clusters cut across suites, so they don't show up in the table above
(counts from the
triage in #12388): **~226** tests failing on `IncrementMetric` (#9003) and
**~47** on the Delta
stats tracker `ClassCastException` when the stats plan can't be offloaded
(#12538).
Related type-support gap: TIMESTAMP_NTZ (#11622), which is one trigger of
#12538.
- [ ] #9003 — IncrementMetric
- [ ] #12538 — Delta stats tracker should fall back instead of throwing
- [ ] #11622 — TIMESTAMP_NTZ support
- [ ] #4963 — Hadoop file system options
- [ ] Triage the `DescribeDeltaHistory`, `ImplicitMergeCasting`,
`RowTrackingBackfill` and
`DataSkipping` clusters and file one issue per root cause
## 2. Native bugs currently worked around in CI
Both of these are *hidden* by the pipeline today, so the workarounds should
be removed together
with the fixes.
- [ ] #12377 — DV bitmap aggregator aborts on an invalid row index.
Currently quarantined by
error signature in `flaky-error-patterns.txt` (2 patterns). Remove
those entries when fixed.
- [ ] #12387 — native memory OOM when reading billions-of-rows tables. Two
`DeletionVectorsSuite` 2B-row tests are force-failed in
`setup-delta.sh` to stop them
OOM-killing the shard. Remove that patch when fixed.
- [ ] Unsupported operations should induce a **fallback** for Delta plans
rather than failing
(raised by @zhztheplayer in #12388; #12538 is the concrete instance)
## 3. Pipeline improvements
- [ ] **Share the Velox native lib across workflows.** `gluten-delta/**` and
`backends-velox/src-delta*/**` match both `velox_backend_x86.yml` and
`delta_spark_ut.yml`,
so those PRs build the native lib twice (~10 min). Suggested by
@philo-he and @zhouyuan.
A cache is preferable to artifact-passing: artifacts are scoped to a
workflow *run* and
there is no cross-workflow `needs`, so the Delta workflow would have
to poll a sibling run
it races with, whereas a cache populated by push-to-`main` has no
ordering problem.
**The key must cover `cpp/**`, the build flags and the base image** —
unlike ccache, where
a bad key only costs a rebuild, a stale hit on a prebuilt lib means
silently testing the
wrong binary. A miss must fall back to building.
- [ ] **Cover more Spark versions.** The workflow already takes
`spark_version` / `delta_ref`
inputs, and `pom.xml` pairs Spark 3.5→Delta 3.3.2, 4.0→4.0.1,
4.1→4.1.0. The blocker is the
baseline: each combination needs its own `known-failures.txt` and a
bootstrap run.
Suggested starting point: **Spark 3.5 as a nightly-only second leg**,
which keeps per-PR
cost unchanged. Requested by @zhouyuan.
- [ ] **Remove the overlapping imported Delta suites.** @malinjawi
identified 8 suites imported
from delta-io/delta (`DeltaSuite`, `DeleteSQLSuite` + DV variants,
`UpdateSQLSuite`,
`DeltaDDLSuite`, `DeltaInsertIntoTableSuite`,
`ClusteredTableClusteringSuite`,
`OptimizeMetadataOnlyDeltaQuerySuite`, `OptimizedWritesSuite`) that
this pipeline now
duplicates — but **only on the Spark 4.1 leg**. They are the only
Delta coverage on Spark
3.5 and 4.0, so this is blocked on the Spark 3.5 leg above. Raised by
@zhztheplayer.
- [ ] **Delta 4.3.0 support.** Attempted during #12388: the bundle is
compiled against Delta
4.1.0 and hits a binary-incompatible change
(`IdentityColumn.logTableWrite`, first param
`Snapshot` → `SnapshotDescriptor`), which `NoSuchMethodError`s on
every write. Needs the
bundle built against 4.3.0.
- [ ] **`run-delta-ci` label opt-in**, if wanted. Dropped in #12388: it
cannot be expressed as a
`paths:` filter, so it would require a gate job running `git diff` on
every PR, and the
label did not exist on the repo. `workflow_dispatch` covers the same
need today.
- [ ] **Drop the upstream cherry-picks** (delta-io/delta#7104 + #7105,
`FileSourceScanExec` →
`FileSourceScanLike`) once `DELTA_REF` moves to a release that
contains them.
`setup-delta.sh` already skips a cherry-pick that is already applied,
so this is cleanup.
## Notes for whoever picks these up
- Refresh the baseline by running **Delta Spark UT (Gluten)** with
`update_baseline=true` and
committing the produced artifact; see
`.github/workflows/util/delta-spark-ut/README.md`.
- A test that fails **consistently** belongs in `known-failures.txt`.
`flaky-tests.txt` /
`flaky-error-patterns.txt` are only for tests that genuinely pass on some
runs and fail on
others — quarantining a deterministic failure hides a real bug.
- The nightly run enforces `fail_on_fixed=true`, so when a fix lands, the
nightly goes red until
the now-passing tests are removed from the baseline. That is the intended
signal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
