This is an automated email from the ASF dual-hosted git repository.
zhouyuan pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 3f1c2f5b47 [GLUTEN-13037][CI] Extract temporary Delta test patches
(#13057)
3f1c2f5b47 is described below
commit 3f1c2f5b47f87482be3cc7ed1f281e63e1e3ed09
Author: Felipe Pessoto <[email protected]>
AuthorDate: Tue Sep 22 23:48:36 2026 -0700
[GLUTEN-13037][CI] Extract temporary Delta test patches (#13057)
---
.github/workflows/delta_spark_ut.yml | 3 +-
.github/workflows/util/delta-spark-ut/README.md | 1 +
...{setup-delta.sh => apply-delta-test-patches.sh} | 129 ++----------------
.../workflows/util/delta-spark-ut/setup-delta.sh | 144 +--------------------
4 files changed, 14 insertions(+), 263 deletions(-)
diff --git a/.github/workflows/delta_spark_ut.yml
b/.github/workflows/delta_spark_ut.yml
index b9d240d8d8..0029c977be 100644
--- a/.github/workflows/delta_spark_ut.yml
+++ b/.github/workflows/delta_spark_ut.yml
@@ -140,7 +140,8 @@ env:
# Memory is per runner and therefore unaffected: each shard is its own job,
so
# it is still 4 forks plus the sbt launcher against the ~16G limit, at ~4G
per
# fork (2G heap + 2G off-heap). That fits because the worst memory hog
- # (DeletionVectorsSuite 2B-row) is force-failed in setup-delta.sh.
+ # (DeletionVectorsSuite 2B-row) is force-failed in
+ # apply-delta-test-patches.sh.
#
# Further sharding has a floor: wall clock cannot drop below the longest
single
# suite, which is ~18 min (DeleteSQLSQLPathBasedDVPredPushOffSuite). At 8
shards
diff --git a/.github/workflows/util/delta-spark-ut/README.md
b/.github/workflows/util/delta-spark-ut/README.md
index be7232fab4..b94c738da5 100644
--- a/.github/workflows/util/delta-spark-ut/README.md
+++ b/.github/workflows/util/delta-spark-ut/README.md
@@ -43,6 +43,7 @@ starts failing** (a regression).
| `run-delta-tests.sh` | The shard step's body: runs `sbt spark/test` (tuned
JVM/heap flags) under a hang watchdog, prints memory forensics, then gates the
results against the baseline via `compare-test-results.py`. |
| `java-test-args.sh` | Shared JVM flags (`--add-opens` + Netty property)
needed to run the suite on JDK 17 with the Gluten bundle. Sourced by
`run-delta-tests.sh` and by local runs. |
| `setup-delta.sh` | Clones Delta, drops in the Gluten bundle, and patches
`DeltaSQLCommandTest`. |
+| `apply-delta-test-patches.sh` | Applies temporary Delta test workarounds:
upstream scan fixes, deterministic Parquet row groups, and 2B-row DV fail-fast
patches. Called by `setup-delta.sh` with `<delta_ref> <delta_dir>`. |
## How the gate works
diff --git a/.github/workflows/util/delta-spark-ut/setup-delta.sh
b/.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh
similarity index 58%
copy from .github/workflows/util/delta-spark-ut/setup-delta.sh
copy to .github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh
index a563bec74f..4fd9c874a6 100755
--- a/.github/workflows/util/delta-spark-ut/setup-delta.sh
+++ b/.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh
@@ -16,114 +16,24 @@
# limitations under the License.
#
-# Prepares a delta-io/delta clone for running its `spark` module tests with the
-# Gluten (Velox) bundle jar on the classpath.
+# Applies temporary Gluten compatibility patches to a delta-io/delta checkout.
#
# Usage:
-# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar>
<gluten_repo_root>
+# apply-delta-test-patches.sh <delta_ref> <delta_dir>
#
-# Arguments:
-# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.2.0)
-# delta_dir - destination directory for the Delta clone
-# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
-# gluten_repo_root - path to the Gluten repository root (used to locate
-#
backends-velox/src-delta40/.../DeltaSQLCommandTest.scala)
+# Remove each patch group when DELTA_REF contains the corresponding upstream
+# fix or Gluten no longer needs the workaround.
#
set -euo pipefail
-if [ "$#" -ne 4 ]; then
- echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar>
<gluten_repo_root>" >&2
+if [ "$#" -ne 2 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir>" >&2
exit 1
fi
DELTA_REF="$1"
DELTA_DIR="$2"
-GLUTEN_BUNDLE_JAR="$3"
-GLUTEN_ROOT="$4"
-
-if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
- echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
- exit 1
-fi
-
-# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
-# rather than maintaining a separate copy. This file is compiled as part of the
-# unified `spark` project's Test scope, which has the Gluten bundle on its
-# classpath (via spark-unified/lib/), so the typed GlutenConfig /
VeloxDeltaConfig
-# imports resolve correctly.
-PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta40/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
-if [ ! -f "$PATCH_SOURCE" ]; then
- echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
- exit 1
-fi
-
-echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
-# init + shallow fetch resolves a tag, branch OR commit SHA in a single path
-# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the
-# destructive `rm -rf "$DELTA_DIR"` it required. `--` terminates options so a
-# DELTA_REF starting with `-` can't be misread as a git flag (this script is
-# workflow_dispatch-runnable with a user-supplied ref).
-#
-# Every step here is idempotent so a local re-run (or a CI re-run on a runner
-# that kept the workspace) resumes instead of dying: `git init` re-initializes
-# an existing repo harmlessly, but `remote add` errors out when `origin`
already
-# exists, so drop it first; and `checkout -f` discards leftovers from a
previous
-# partial run. Nothing worth keeping exists here yet -- the bundle jar and the
-# source patches below are applied *after* this block.
-git init -q "$DELTA_DIR"
-git -C "$DELTA_DIR" remote remove origin 2>/dev/null || true
-git -C "$DELTA_DIR" remote add origin https://github.com/delta-io/delta.git
-git -C "$DELTA_DIR" fetch -q --depth 1 origin -- "$DELTA_REF"
-git -C "$DELTA_DIR" checkout -qf FETCH_HEAD
-git -C "$DELTA_DIR" --no-pager log -1 --oneline
-echo "::endgroup::"
-
-echo "::group::Injecting Gluten bundle jar onto the spark project's TEST
classpath"
-# The Gluten bundle jar must be on the spark project's TEST runtime classpath
-# (so DeltaSQLCommandTest can load org.apache.gluten.GlutenPlugin by name) but
-# NOT on the COMPILE classpath of `sparkV1`, which is the project that holds
-# Delta's main sources. The bundle's transitive contents include extra symbols
-# under `org.apache.spark.sql` that collide with Delta's main sources -- e.g.
-# MergeOutputGeneration.scala imports both `org.apache.spark.sql._` and
-# `org.apache.spark.sql.delta.ClassicColumnConversions._`, and would then fail
-# with `reference to expression is ambiguous`.
-#
-# sbt auto-scans `<baseDirectory>/lib` via `unmanagedBase`. Two relevant
-# projects in Delta v4.2.0 have a `lib/` baseDirectory:
-# - sparkV1: `project in file("spark")` -> spark/lib
-# - spark : `project in file("spark-unified")` -> spark-unified/lib
-# unmanagedJars are project-scoped (NOT inherited by dependents), so dropping
-# the bundle into spark-unified/lib/ adds it to the unified `spark` project's
-# Compile *and* Test classpaths -- but NOT to sparkV1's. That's exactly what
-# we want:
-# * sparkV1/Compile sees ONLY Delta's regular deps -> Delta main compiles.
-# * spark/Test/fullClasspath sees the bundle -> tests load GlutenPlugin.
-# (Verified empirically: with bundle only in spark-unified/lib/, sbt's
-# `show sparkV1/Compile/dependencyClasspath` excludes the bundle and
-# `show spark/Test/fullClasspath` includes it.)
-#
-# We deliberately do NOT also drop the bundle into spark/lib/, which is what
-# caused the previous compile failure: spark/lib/ is sparkV1's unmanagedBase,
-# and putting the bundle there would re-introduce the ambiguity errors.
-SPARK_UNIFIED_LIB="$DELTA_DIR/spark-unified/lib"
-mkdir -p "$SPARK_UNIFIED_LIB"
-cp "$GLUTEN_BUNDLE_JAR" "$SPARK_UNIFIED_LIB/gluten-velox-bundle.jar"
-ls -lh "$SPARK_UNIFIED_LIB"
-echo "::endgroup::"
-
-echo "::group::Patching DeltaSQLCommandTest to enable Gluten plugin"
-TARGET="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
-if [ ! -f "$TARGET" ]; then
- echo "Expected file not found in Delta clone: $TARGET" >&2
- echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
- exit 1
-fi
-cp "$PATCH_SOURCE" "$TARGET"
-echo "Patched $TARGET"
-echo "--- diff vs. upstream ---"
-git -C "$DELTA_DIR" --no-pager diff --
"spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
|| true
-echo "::endgroup::"
# Delta's tests collect file-source scans by matching the concrete
# `FileSourceScanExec` case class; Gluten offloads the scan to
@@ -261,34 +171,13 @@ sed -i 's#number of rows from tables of 2B rows with
DVs") {#&\n fail("[Glu
INJECTED=$(grep -c "Gluten CI] Force-failed" "$DVS" || true)
if [ "$INJECTED" -ne 2 ]; then
echo "ERROR: expected to force-fail 2 DeletionVectorsSuite tests but
injected ${INJECTED}." >&2
- echo "Their test names likely changed in Delta ref '${DELTA_REF}'; update
setup-delta.sh." >&2
+ echo "Their test names likely changed in Delta ref '${DELTA_REF}'; update
apply-delta-test-patches.sh." >&2
exit 1
fi
echo "Force-failed 2 DeletionVectorsSuite 2B-row tests (read + delete)."
git -C "$DELTA_DIR" --no-pager diff --
"spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala"
|| true
echo "::endgroup::"
-echo "::group::Disabling Delta scalastyle HeaderMatchesChecker"
-# Our reused DeltaSQLCommandTest carries Gluten's ASF-only license header,
which
-# does not match Delta's HeaderMatchesChecker regex (the regex expects either a
-# Delta copyright block, or the ASF header followed by a Spark-modifications
-# block and the Delta copyright block). HeaderMatchesChecker is a file-level
-# checker that does NOT honor `// scalastyle:off` directives, so we instead
-# disable it globally in Delta's shared scalastyle-config.xml. The config is
-# applied via `ThisBuild / scalastyleConfig` in project/Checkstyle.scala, so a
-# single edit covers every sbt sub-project.
-SCALASTYLE_CONFIG="$DELTA_DIR/scalastyle-config.xml"
-if [ ! -f "$SCALASTYLE_CONFIG" ]; then
- echo "Expected scalastyle config not found: $SCALASTYLE_CONFIG" >&2
- exit 1
-fi
-sed -i \
- 's|<check level="error" class="org.scalastyle.file.HeaderMatchesChecker"
enabled="true">|<check level="error"
class="org.scalastyle.file.HeaderMatchesChecker" enabled="false">|' \
- "$SCALASTYLE_CONFIG"
-if ! grep -q '<check level="error"
class="org.scalastyle.file.HeaderMatchesChecker" enabled="false">'
"$SCALASTYLE_CONFIG"; then
- echo "Failed to disable HeaderMatchesChecker in $SCALASTYLE_CONFIG" >&2
- grep -n 'HeaderMatchesChecker' "$SCALASTYLE_CONFIG" >&2 || true
- exit 1
-fi
-echo "Disabled HeaderMatchesChecker in $SCALASTYLE_CONFIG"
+echo "::group::Resulting temporary Delta test source diff"
+git -C "$DELTA_DIR" --no-pager diff HEAD -- "spark/src/test"
echo "::endgroup::"
diff --git a/.github/workflows/util/delta-spark-ut/setup-delta.sh
b/.github/workflows/util/delta-spark-ut/setup-delta.sh
index a563bec74f..629f758678 100755
--- a/.github/workflows/util/delta-spark-ut/setup-delta.sh
+++ b/.github/workflows/util/delta-spark-ut/setup-delta.sh
@@ -125,148 +125,8 @@ echo "--- diff vs. upstream ---"
git -C "$DELTA_DIR" --no-pager diff --
"spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
|| true
echo "::endgroup::"
-# Delta's tests collect file-source scans by matching the concrete
-# `FileSourceScanExec` case class; Gluten offloads the scan to
-# DeltaScanTransformer, a `FileSourceScanLike` sibling, so those matches miss
-# (`scala.MatchError: List()`, empty partition filters, broken column-pruning /
-# scan-metric checks across many suites). delta-io/delta#7104 and #7105 widen
the
-# matches to the shared `FileSourceScanLike` interface that both the vanilla
and
-# Gluten scans implement (behavior-preserving for vanilla). Both are merged
-# upstream but land after the pinned DELTA_REF (v4.2.0), so apply them here;
once
-# DELTA_REF includes a fix, cherry_pick_delta_fix detects it and skips (see
below).
-#
-# Depth-2 fetch brings each fix commit and its parent, which cherry-pick needs
to
-# diff against (a depth-1 fetch grafts the parent away); `-n` stages the change
-# without requiring a committer identity.
-cherry_pick_delta_fix() {
- local sha="$1" pr="$2"
- git -C "$DELTA_DIR" fetch --quiet --depth 2 origin "$sha"
- echo "Cherry-picking delta-io/delta${pr}"
- if git -C "$DELTA_DIR" cherry-pick -n "$sha"; then
- return 0
- fi
- # The cherry-pick did not apply. The usual cause is that the pinned DELTA_REF
- # already contains this fix (e.g. after a version bump), which makes the
patch
- # empty/conflicting and would -- under `set -e` -- abort the whole setup. We
- # can't use ancestry to tell "already contained" from a genuine conflict here
- # (the clone is shallow, so `merge-base --is-ancestor` can't see past the
graft),
- # so recover the exact paths this fix touches -- leaving other setup such as
the
- # DeltaSQLCommandTest patch intact -- and continue. This is self-correcting:
if
- # the fix is genuinely still needed, the FileSourceScanLike failures it
prevents
- # resurface as gate regressions rather than being hidden by a hard abort
here.
- echo "Cherry-pick of delta-io/delta${pr} did not apply cleanly" \
- "(most likely already contained in ${DELTA_REF}); skipping it."
- local f
- while IFS= read -r f; do
- [ -n "$f" ] || continue
- git -C "$DELTA_DIR" reset -q -- "$f" 2>/dev/null || true
- git -C "$DELTA_DIR" checkout -q -- "$f" 2>/dev/null || true
- done < <(git -C "$DELTA_DIR" diff-tree --no-commit-id --name-only -r "$sha")
- # Clear any leftover sequencer state (harmless if none exists).
- git -C "$DELTA_DIR" cherry-pick --quit 2>/dev/null || true
- return 0
-}
-
-echo "::group::Cherry-picking upstream Delta FileSourceScanLike test fixes"
-cherry_pick_delta_fix 46bd45d57eadd7e528002a0ae7bd36ce5a456eca "#7104
(ScanReportHelper.collectScans)"
-cherry_pick_delta_fix 959e00e15f41f56afc1c9bb95d160c55c6dc7068 "#7105 (9 more
test suites)"
-echo "::endgroup::"
-
-echo "::group::Capping DeltaParquetFileFormat fixture row groups by row count"
-# DeltaParquetFileFormatSuite generates one 20,000-row Parquet file and sets a
-# 50 KiB block size to ensure that it contains multiple row groups. Velox sizes
-# row groups by buffered bytes after writing each input batch. Because this
-# fixture arrives in one batch, lowering the byte threshold cannot split it.
-# Scope Gluten's native row-count limit around the fixture write so Arrow
splits
-# the 20,000 rows deterministically while keeping the native write path
enabled.
-DPFFS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala"
-if [ ! -f "$DPFFS" ]; then
- echo "Expected file not found in Delta clone: $DPFFS" >&2
- echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
>&2
- exit 1
-fi
-if ! sed 's/^__BLANK_CONTEXT__$/ /' <<'PATCH' | git -C "$DELTA_DIR" apply -
-diff --git
a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
----
a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
-+++
b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
-@@ -68,9 +68,11 @@ trait DeltaParquetFileFormatSuiteBase
- protected def generateData(tablePath: String): Unit = {
- // This is to generate a Parquet file with two row groups
- hadoopConf().set("parquet.block.size", (1024 * 50).toString)
-__BLANK_CONTEXT__
- // Keep the number of partitions to 1 to generate a single Parquet data
file
- val df = Seq.range(0, 20000).toDF().repartition(1)
-- df.write.format("delta").mode("append").save(tablePath)
-+ withSQLConf("spark.gluten.sql.native.parquet.write.blockRows" -> "10000")
{
-+ df.write.format("delta").mode("append").save(tablePath)
-+ }
-__BLANK_CONTEXT__
- // Set DFS block size to be less than Parquet rowgroup size, to allow
-PATCH
-then
- echo "ERROR: DeltaParquetFileFormat fixture patch did not apply." >&2
- echo "The patch expects the Delta v4.2.0 generateData fixture shape;" \
- "ref '${DELTA_REF}' must remain source-compatible." >&2
- exit 1
-fi
-ROW_CAP_SCOPES=$(
- grep -Fxc \
- ' withSQLConf("spark.gluten.sql.native.parquet.write.blockRows" ->
"10000") {' \
- "$DPFFS" || true
-)
-if [ "$ROW_CAP_SCOPES" -ne 1 ]; then
- echo "ERROR: expected exactly one native Parquet row-count scope;" \
- "found ${ROW_CAP_SCOPES}." >&2
- echo "DeltaParquetFileFormatSuite may have changed in Delta ref
'${DELTA_REF}'." >&2
- exit 1
-fi
-echo "Capped DeltaParquetFileFormat fixture row groups at 10,000 rows."
-git -C "$DELTA_DIR" --no-pager diff -- \
-
"spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala"
|| true
-echo "::endgroup::"
-
-echo "::group::Force-failing memory-hog DeletionVectorsSuite 2B-row tests"
-# Two DeletionVectorsSuite tests read from / delete from a 2-billion-row table.
-# Under the Gluten Velox bundle they balloon the forked test JVM to ~13G of
-# NATIVE memory (row-index materialization) and the kernel/cgroup OOM-kills it.
-# The dead fork then wedges sbt, hanging the whole shard until the workflow's
-# hang-watchdog dumps threads and kills it (~16 min wasted, and every suite
-# QUEUED AFTER it in that fork is skipped) -- see delta_spark_ut.yml.
-#
-# Rather than silently `ignore` these (easy to forget), we make them FAIL FAST
-# with a clear message: the gap stays visible in the test reports / baseline
-# until the native memory blow-up is fixed, at which point this patch should be
-# removed. NOTE: making the suite complete also un-skips the rest of the
shard's
-# suite queue, so the known-failures baseline must be refreshed after this.
-#
-# ORDER MATTERS: keep this sed AFTER the cherry-picks above. #7105 also edits
-# DeletionVectorsSuite.scala, and git cherry-pick aborts (exit 128) when the
work
-# tree has uncommitted edits to a file it touches. It also relies on the clone
-# step's `checkout -f`: the sed appends after the declaration line, so without
-# that per-run reset a re-run injects duplicate `fail` lines and trips the
-# INJECTED != 2 check below.
-DVS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala"
-if [ ! -f "$DVS" ]; then
- echo "Expected file not found in Delta clone: $DVS" >&2
- echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
>&2
- exit 1
-fi
-# Inject `fail(...)` as the first statement of each test body (the line ending
-# in `) {`). Delta sets no -Xfatal-warnings / dead-code warning, so the now-
-# unreachable original body compiles fine. Keep each injected line <100 chars:
-# Delta's scalastyle enforces a 100-char line length on test sources. The full
-# rationale lives in this comment, so the in-test message stays terse.
-sed -i 's#huge table: read from tables of 2B rows with existing DV of many
zeros") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV read OOMs the test
JVM; see setup-delta.sh")#' "$DVS"
-sed -i 's#number of rows from tables of 2B rows with DVs") {#&\n
fail("[Gluten CI] Force-failed: 2B-row DV delete OOMs the test JVM; see
setup-delta.sh")#' "$DVS"
-INJECTED=$(grep -c "Gluten CI] Force-failed" "$DVS" || true)
-if [ "$INJECTED" -ne 2 ]; then
- echo "ERROR: expected to force-fail 2 DeletionVectorsSuite tests but
injected ${INJECTED}." >&2
- echo "Their test names likely changed in Delta ref '${DELTA_REF}'; update
setup-delta.sh." >&2
- exit 1
-fi
-echo "Force-failed 2 DeletionVectorsSuite 2B-row tests (read + delete)."
-git -C "$DELTA_DIR" --no-pager diff --
"spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala"
|| true
-echo "::endgroup::"
+PATCH_SCRIPT="$(cd "$(dirname "${BASH_SOURCE[0]}")" &&
pwd)/apply-delta-test-patches.sh"
+bash "$PATCH_SCRIPT" "$DELTA_REF" "$DELTA_DIR"
echo "::group::Disabling Delta scalastyle HeaderMatchesChecker"
# Our reused DeltaSQLCommandTest carries Gluten's ASF-only license header,
which
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
