This is an automated email from the ASF dual-hosted git repository.
philo-he pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new e681b63d47 [MINOR][VL] Configure Gluten for all Delta Spark test
contexts (#13131)
e681b63d47 is described below
commit e681b63d47aa5bffdc8db4eb0ee37f43fd8b5ced
Author: Felipe Pessoto <[email protected]>
AuthorDate: Thu Oct 29 14:20:52 2026 -0700
[MINOR][VL] Configure Gluten for all Delta Spark test contexts (#13131)
---
.github/workflows/delta_spark_ut.yml | 11 ++--
.github/workflows/util/delta-spark-ut/README.md | 52 +++++++++++++++---
.../delta-spark-ut/apply-delta-test-patches.sh | 5 +-
.../util/delta-spark-ut/java-test-args.sh | 27 +++++++++-
.../util/delta-spark-ut/known-failures.txt | 19 -------
.../patches/mock-s3-fixture-writer.patch | 24 +++++++++
.../util/delta-spark-ut/run-delta-tests.sh | 8 +--
.../workflows/util/delta-spark-ut/setup-delta.sh | 63 +++-------------------
8 files changed, 112 insertions(+), 97 deletions(-)
diff --git a/.github/workflows/delta_spark_ut.yml
b/.github/workflows/delta_spark_ut.yml
index cc06530d0b..3917605354 100644
--- a/.github/workflows/delta_spark_ut.yml
+++ b/.github/workflows/delta_spark_ut.yml
@@ -23,9 +23,9 @@
# 3. Clones delta-io/delta at the requested release tag (default: the Delta
# release the bundle is compiled against, `delta4.version` in pom.xml),
# drops the bundle jar into `spark-unified/lib/` only (NOT `spark/lib/`
-# -- see setup-delta.sh for the unmanagedJars scoping rationale),
-# patches Delta's `DeltaSQLCommandTest` to register the Gluten plugin,
-# and runs `sbt spark/test` sharded across the matrix.
+# -- see setup-delta.sh for the unmanagedJars scoping rationale), applies
+# temporary compatibility patches, and runs `sbt spark/test` sharded
+# across the matrix with JVM-wide Gluten defaults from java-test-args.sh.
#
# Limited to Velox + x86 to keep the matrix simple, per the pipeline's purpose
# of validating Gluten changes against the latest Delta release.
@@ -464,8 +464,7 @@ jobs:
bash
"$GITHUB_WORKSPACE/.github/workflows/util/delta-spark-ut/setup-delta.sh" \
"${{ steps.resolve.outputs.delta_ref }}" \
"$GITHUB_WORKSPACE/delta" \
- "$GLUTEN_JAR" \
- "$GITHUB_WORKSPACE"
+ "$GLUTEN_JAR"
- name: Run Delta spark module tests (shard ${{ matrix.shard }} / ${{
env.DELTA_NUM_SHARDS }})
env:
@@ -477,7 +476,7 @@ jobs:
# Required by Delta to enable testing-only code paths
# (see delta build.sbt: "Test / envVars += DELTA_TESTING -> 1").
DELTA_TESTING: '1'
- # NOTE: the Gluten/JDK17 test JVM flags (JAVA_TOOL_OPTIONS) live in
+ # NOTE: Gluten defaults and JDK17 JVM flags (JAVA_TOOL_OPTIONS) live
in
# util/delta-spark-ut/java-test-args.sh, which run-delta-tests.sh
sources,
# so CI and local dev runs share one definition.
run: |
diff --git a/.github/workflows/util/delta-spark-ut/README.md
b/.github/workflows/util/delta-spark-ut/README.md
index 78c4bb4d1a..cecee7f3ea 100644
--- a/.github/workflows/util/delta-spark-ut/README.md
+++ b/.github/workflows/util/delta-spark-ut/README.md
@@ -41,10 +41,10 @@ starts failing** (a regression).
| `flaky-error-patterns.txt` | Quarantine list by error signature: regex
patterns matched against a failure's text, for bugs that surface on a different
test each run (e.g. the native DV bitmap row-index error). |
| `compare-test-results.py` | Parses the JUnit XML from `sbt spark/test` and
gates / seeds / aggregates against the baseline. Standard-library only. |
| `run-delta-tests.sh` | The shard step's body: runs `sbt spark/test` (tuned
JVM/heap flags) under a hang watchdog, prints memory forensics, then gates the
results against the baseline via `compare-test-results.py`. |
-| `java-test-args.sh` | Shared JVM flags (`--add-opens` + Netty property)
needed to run the suite on JDK 17 with the Gluten bundle. Sourced by
`run-delta-tests.sh` and by local runs. |
-| `setup-delta.sh` | Clones Delta, drops in the Gluten bundle, and patches
`DeltaSQLCommandTest`. |
+| `java-test-args.sh` | Shared Gluten Spark defaults and JVM flags
(`--add-opens` + Netty property). Sourced by `run-delta-tests.sh` and by local
runs. |
+| `setup-delta.sh` | Clones Delta, drops in the Gluten bundle, and applies
temporary compatibility patches. Takes `<delta_ref> <delta_dir>
<gluten_bundle_jar>`. |
| `apply-delta-test-patches.sh` | Cherry-picks upstream scan fixes, applies
the ordered local patch list, and verifies the resulting test sources. Called
by `setup-delta.sh` with `<delta_ref> <delta_dir>`. |
-| `patches/*.patch` | Separate patches for CDF plan assertions, the two
Parquet row-group fixtures, and 2B-row DV fail-fast behavior. Each patch
documents its rationale and removal condition. |
+| `patches/*.patch` | Separate patches for CDF plan assertions, mock-S3
fixture writes, the two Parquet row-group fixtures, and 2B-row DV fail-fast
behavior. Each patch documents its rationale and removal condition. |
## Temporary Delta test patches
@@ -85,6 +85,41 @@ it when the display gap is fixed or Delta's assertions no
longer require the
workaround. The affected cases are no longer in `known-failures.txt`, so their
failures remain enforced by the gate.
+## Gluten configuration for every Spark context
+
+`java-test-args.sh` exports the Gluten plugin, columnar shuffle manager, and
+off-heap memory settings (enabled, 2 GiB) as JVM system properties through
+`JAVA_TOOL_OPTIONS`. `SparkConf` reads these properties as defaults, so they
+also reach suites using plain `SharedSparkSession`, `TestHive`, or their own
+`SparkSession` / `SparkContext` instances instead of `DeltaSQLCommandTest`.
+
+This is required even when those suites do not explicitly select Gluten's
+cache serializer: Spark retains the first cache serializer across contexts
+in the same JVM. A later context without the Gluten executor plugin would
+therefore use the native serializer without initializing its task resource
+registry, causing `TaskResourceRegistry is not initialized`.
+
+The remaining defaults formerly injected through `DeltaSQLCommandTest` also
+apply to every suite:
+
+| Setting | Default |
+|---|---|
+| `spark.default.parallelism` | `1` |
+| `spark.sql.shuffle.partitions` | `5` |
+| `spark.unsafe.exceptionOnMemoryLeak` | `true` |
+| `spark.sql.ansi.enabled` | `false` |
+| `spark.gluten.sql.ansiFallback.enabled` | `false` |
+| `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite` | `true` |
+| `spark.databricks.delta.snapshotPartitions` | `2` |
+| `spark.gluten.sql.fallbackUnexpectedMetadataParquet` | `true` |
+
+Suites can still override these defaults through their Spark configuration.
+In particular, ANSI-specific tests can enable ANSI explicitly.
+
+Delta extensions and the Delta catalog remain suite-specific rather than
+becoming JVM defaults, because some suites test behavior when those settings
+are absent.
+
## How the gate works
Each test shard:
@@ -280,10 +315,10 @@ python3
.github/workflows/util/delta-spark-ut/compare-test-results.py \
## Running the suite locally
-`sbt spark/test` needs extra JDK-17 JVM flags to run the Delta suite against
the
-Gluten bundle (`--add-opens` + the Netty reflection property). CI and local
runs
-share one definition in `java-test-args.sh` — `source` it before invoking sbt
so
-the flags reach the sbt launcher and the forked test JVM:
+`sbt spark/test` needs Gluten Spark defaults and extra JDK-17 JVM flags
+(`--add-opens` + the Netty reflection property). CI and local runs share one
+definition in `java-test-args.sh` — `source` it before invoking sbt so the
+configuration reaches the sbt launcher and every forked test JVM:
```bash
# from the Delta clone prepared by setup-delta.sh (which has the Gluten
bundle):
@@ -293,3 +328,6 @@ source
<gluten>/.github/workflows/util/delta-spark-ut/java-test-args.sh
`run-delta-tests.sh` sources the same file, so CI and local runs use identical
flags.
+
+Existing `JAVA_TOOL_OPTIONS` are preserved after the shared defaults, so
+caller-supplied `-D` properties take precedence.
diff --git a/.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh
b/.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh
index b1bc85c085..a000eced1b 100755
--- a/.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh
+++ b/.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh
@@ -61,8 +61,8 @@ cherry_pick_delta_fix() {
# empty/conflicting and would -- under `set -e` -- abort the whole setup. We
# can't use ancestry to tell "already contained" from a genuine conflict here
# (the clone is shallow, so `merge-base --is-ancestor` can't see past the
graft),
- # so recover the exact paths this fix touches -- leaving other setup such as
the
- # DeltaSQLCommandTest patch intact -- and continue. This is self-correcting:
if
+ # so recover only the paths this fix touches and continue, preserving other
+ # setup changes. This is self-correcting: if
# the fix is genuinely still needed, the FileSourceScanLike failures it
prevents
# resurface as gate regressions rather than being hidden by a hard abort
here.
echo "Cherry-pick of delta-io/delta${pr} did not apply cleanly" \
@@ -88,6 +88,7 @@ echo "::group::Applying temporary Delta test patches"
# for the files they touch. Patch paths are relative to this script, not Delta.
for patch in \
cdf-pushed-filter-assertions.patch \
+ mock-s3-fixture-writer.patch \
parquet-fixture-row-groups.patch \
dv-predicate-pushdown-row-groups.patch \
dv-2b-row-fail-fast.patch; do
diff --git a/.github/workflows/util/delta-spark-ut/java-test-args.sh
b/.github/workflows/util/delta-spark-ut/java-test-args.sh
index 94b3246a9f..a985dc1fa7 100755
--- a/.github/workflows/util/delta-spark-ut/java-test-args.sh
+++ b/.github/workflows/util/delta-spark-ut/java-test-args.sh
@@ -35,11 +35,22 @@
# the sbt LAUNCHER JVM) so the flags reach BOTH the launcher and the forked
test JVM
# -- the forked child inherits the parent env.
#
+# SparkConf also reads spark.* system properties as defaults. Keep all Gluten
+# test defaults here instead of replacing DeltaSQLCommandTest, so suites using
+# plain SharedSparkSession, TestHive or their own SparkSession builders inherit
+# the same settings. Those contexts must initialize Gluten too: Spark retains
+# its cache serializer across contexts in the same JVM, but the native
+# serializer needs the executor plugin to initialize TaskResources in each
task.
+#
+# Keep Delta's extensions and catalog suite-specific: some tests deliberately
+# omit them to check Delta's configuration errors.
+#
# NOTE: no -Xmx here on purpose. JAVA_TOOL_OPTIONS is processed BEFORE the JVM
# command line, so Delta's own -Xmx (build.sbt) would win; run-delta-tests.sh
bumps
# the forked-test-JVM heap via `set spark / Test / javaOptions ++= ...`
instead.
-export JAVA_TOOL_OPTIONS="${JAVA_TOOL_OPTIONS:+${JAVA_TOOL_OPTIONS} }\
+# Keep caller-provided -D properties last so they override these defaults.
+export JAVA_TOOL_OPTIONS="\
-XX:+IgnoreUnrecognizedVMOptions \
--add-opens=java.base/java.lang=ALL-UNNAMED \
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED \
@@ -57,4 +68,16 @@ export
JAVA_TOOL_OPTIONS="${JAVA_TOOL_OPTIONS:+${JAVA_TOOL_OPTIONS} }\
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED \
-Djdk.reflect.useDirectMethodHandle=false \
-Dio.netty.tryReflectionSetAccessible=true \
--Dfile.encoding=UTF-8"
+-Dfile.encoding=UTF-8 \
+-Dspark.plugins=org.apache.gluten.GlutenPlugin \
+-Dspark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+-Dspark.memory.offHeap.enabled=true \
+-Dspark.memory.offHeap.size=2g \
+-Dspark.default.parallelism=1 \
+-Dspark.sql.shuffle.partitions=5 \
+-Dspark.unsafe.exceptionOnMemoryLeak=true \
+-Dspark.sql.ansi.enabled=false \
+-Dspark.gluten.sql.ansiFallback.enabled=false \
+-Dspark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true \
+-Dspark.databricks.delta.snapshotPartitions=2 \
+-Dspark.gluten.sql.fallbackUnexpectedMetadataParquet=true${JAVA_TOOL_OPTIONS:+
${JAVA_TOOL_OPTIONS}}"
diff --git a/.github/workflows/util/delta-spark-ut/known-failures.txt
b/.github/workflows/util/delta-spark-ut/known-failures.txt
index 9825b11cb3..5b941b86a5 100644
--- a/.github/workflows/util/delta-spark-ut/known-failures.txt
+++ b/.github/workflows/util/delta-spark-ut/known-failures.txt
@@ -20,9 +20,6 @@
# Velox forks x ~4G run close to the ~16G runner limit) -- so a baseline
captured
# at a different parallelism spuriously flags them as regressions.
# ---------------------------------------------------------------------------
-io.delta.sql.DeltaExtensionAndCatalogSuite#activate Delta SQL parser using SQL
conf
-io.delta.sql.DeltaExtensionAndCatalogSuite#activate Delta SQL parser using
withExtensions
-io.delta.sql.JavaDeltaSparkSessionExtensionSuite#testSQLConf
io.delta.tables.DeltaTableHadoopOptionsSuite#delete - with filesystem options
io.delta.tables.DeltaTableHadoopOptionsSuite#details - with filesystem options.
io.delta.tables.DeltaTableHadoopOptionsSuite#forPath - with filesystem options
@@ -63,9 +60,7 @@
org.apache.spark.sql.delta.CloneTableSQLWithCatalogOwnedBatch2Suite#shallow clon
org.apache.spark.sql.delta.CloneTableScalaDeletionVectorSuite#shallow clone
across file systems
org.apache.spark.sql.delta.CloneTableScalaSuite#shallow clone across file
systems
org.apache.spark.sql.delta.ConvertToDeltaSQLSuite#CONVERT keeps NaN rows for
finite-threshold predicate (NaN-excluding writer)
-org.apache.spark.sql.delta.ConvertToDeltaSQLSuite#external tables use correct
path scheme
org.apache.spark.sql.delta.ConvertToDeltaScalaSuite#CONVERT keeps NaN rows for
finite-threshold predicate (NaN-excluding writer)
-org.apache.spark.sql.delta.ConvertToDeltaScalaSuite#external tables use
correct path scheme
org.apache.spark.sql.delta.DeleteMetricsSuite#delete-metrics: delete one row
per file - Partitioned = false, cdfEnabled = false
org.apache.spark.sql.delta.DeleteMetricsSuite#delete-metrics: delete one row
per file - Partitioned = false, cdfEnabled = true
org.apache.spark.sql.delta.DeltaCDCStreamDeletionVectorSuite#cdc streams with
noop merge
@@ -109,8 +104,6 @@ org.apache.spark.sql.delta.DeltaSuite#SC-8810: skipping
deleted file still throw
org.apache.spark.sql.delta.DeltaSuite#all operations with special characters
in path
org.apache.spark.sql.delta.DeltaSuite#deleted files cause failure by default
org.apache.spark.sql.delta.DeltaTableCreationSuite#Default column values:
CONVERT TO DELTA keeps EXISTS_DEFAULT
-org.apache.spark.sql.delta.DeltaUpdateCatalogSuite#convert to delta with
partitioning change
-org.apache.spark.sql.delta.DeltaUpdateCatalogSuite#partitioned convert to
delta with schema change
org.apache.spark.sql.delta.DeltaVacuumSuite#vacuum for cdc - delete tombstones
org.apache.spark.sql.delta.DeltaVacuumSuite#vacuum for cdc - update/merge
org.apache.spark.sql.delta.DeltaWithCatalogOwnedBatch100Suite#SC-8810: skip
deleted file
@@ -512,15 +505,3 @@
org.apache.spark.sql.delta.typewidening.TypeWideningInsertSchemaEvolutionBasicSu
org.apache.spark.sql.delta.typewidening.TypeWideningInsertSchemaEvolutionBasicSuite#INSERT
- automatic type widening DATE -> TIMESTAMP_NTZ
org.apache.spark.sql.delta.typewidening.TypeWideningMergeIntoSchemaEvolutionSuite#MERGE
- automatic type widening DATE -> TIMESTAMP_NTZ
org.apache.spark.sql.delta.typewidening.TypeWideningTableFeatureAdvancedSuite#drop
feature after type change DATE -> TIMESTAMP_NTZ
-org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap
groupBy aggregate no duplicates - Native
-org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap
groupBy aggregate no duplicates - Portable
-org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap
groupBy aggregate no duplicates - invalid Int ids - Native
-org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap
groupBy aggregate no duplicates - invalid Int ids - Portable
-org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap
groupBy aggregate no duplicates - invalid unsigned Int ids - Native
-org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap
groupBy aggregate no duplicates - invalid unsigned Int ids - Portable
-org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap
groupBy aggregate with duplicates - Native
-org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap
groupBy aggregate with duplicates - Portable
-org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap
groupBy aggregate with duplicates - invalid Int ids - Native
-org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap
groupBy aggregate with duplicates - invalid Int ids - Portable
-org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap
groupBy aggregate with duplicates - invalid unsigned Int ids - Native
-org.apache.spark.sql.delta.util.BitmapAggregatorE2ESuite#DataFrame bitmap
groupBy aggregate with duplicates - invalid unsigned Int ids - Portable
diff --git
a/.github/workflows/util/delta-spark-ut/patches/mock-s3-fixture-writer.patch
b/.github/workflows/util/delta-spark-ut/patches/mock-s3-fixture-writer.patch
new file mode 100644
index 0000000000..fac9002210
--- /dev/null
+++ b/.github/workflows/util/delta-spark-ut/patches/mock-s3-fixture-writer.patch
@@ -0,0 +1,24 @@
+Subject: [PATCH] Use Spark's writer for the Hadoop-only mock S3 fixture
+
+The path-scheme conversion test maps s3: paths to local files through a
+Hadoop FileSystem mock that Velox's native writer cannot use.
+
+Use Spark's writer only for the preparatory CREATE TABLE USING parquet.
+Keep Gluten enabled for the subsequent Delta conversion and preserve the
+original path-scheme assertion. Remove this workaround when the fixture no
+longer relies on the Hadoop-only mock filesystem.
+
+diff --git
a/spark/src/test/scala/org/apache/spark/sql/delta/ConvertToDeltaSuiteBase.scala
b/spark/src/test/scala/org/apache/spark/sql/delta/ConvertToDeltaSuiteBase.scala
+---
a/spark/src/test/scala/org/apache/spark/sql/delta/ConvertToDeltaSuiteBase.scala
++++
b/spark/src/test/scala/org/apache/spark/sql/delta/ConvertToDeltaSuiteBase.scala
+@@ -1450,7 +1450,9 @@
+ withTempDir { dir =>
+ withTable("externalTable") {
+ withSQLConf(("fs.s3.impl",
classOf[S3LikeLocalFileSystem].getCanonicalName)) {
+- sql(s"CREATE TABLE externalTable USING parquet LOCATION 's3://$dir'
AS SELECT 1")
++ withSQLConf("spark.gluten.sql.native.writer.enabled" -> "false") {
++ sql(s"CREATE TABLE externalTable USING parquet LOCATION
's3://$dir' AS SELECT 1")
++ }
+
+ // Ideally we would test a successful conversion with a remote
filesystem, but there's
+ // no good way to set one up in unit tests. So instead we delete
the data, and let the
diff --git a/.github/workflows/util/delta-spark-ut/run-delta-tests.sh
b/.github/workflows/util/delta-spark-ut/run-delta-tests.sh
index a7384097ca..982387806f 100755
--- a/.github/workflows/util/delta-spark-ut/run-delta-tests.sh
+++ b/.github/workflows/util/delta-spark-ut/run-delta-tests.sh
@@ -45,8 +45,8 @@ if command -v getent > /dev/null; then
HOME="$(getent passwd "$(id -u)" | cut -d: -f6)"
export HOME
fi
-# Gluten/JDK17 test JVM flags (--add-opens + Netty property), shared with local
-# dev runs. Sets JAVA_TOOL_OPTIONS so it reaches the sbt launcher + forked
JVMs.
+# Gluten defaults and JDK17 test JVM flags, shared with local dev runs.
+# Sets JAVA_TOOL_OPTIONS so they reach the sbt launcher + forked JVMs.
# shellcheck source=./java-test-args.sh
source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/java-test-args.sh"
cd "$GITHUB_WORKSPACE/delta"
@@ -69,8 +69,8 @@ chmod +x build/sbt
# * Forked test JVM: -Xmx2G via the `set ... Test / javaOptions` command
# below. Delta caps its fork at -Xmx1024m in build.sbt; `++=` appends
# so our -Xmx2G comes last and wins. Gluten offloads data to Velox
-# off-heap (capped at 2g via spark.memory.offHeap.size in the patched
-# DeltaSQLCommandTest), so the fork's heap need is modest. A larger
+# off-heap (capped at 2g via spark.memory.offHeap.size in java-test-args.sh),
+# so the fork's heap need is modest. A larger
# fork heap pushed the cgroup peak past the ~16G OOM threshold and the
# kernel OOM-killed the fork mid-shard (no hs_err), wedging sbt -- 2G
# keeps headroom. Keep heap-dump-on-OOM so a real >2G heap OOM is
diff --git a/.github/workflows/util/delta-spark-ut/setup-delta.sh
b/.github/workflows/util/delta-spark-ut/setup-delta.sh
index 7eb580b6a8..f7905174ed 100755
--- a/.github/workflows/util/delta-spark-ut/setup-delta.sh
+++ b/.github/workflows/util/delta-spark-ut/setup-delta.sh
@@ -17,47 +17,34 @@
#
# Prepares a delta-io/delta clone for running its `spark` module tests with the
-# Gluten (Velox) bundle jar on the classpath.
+# Gluten (Velox) bundle jar on the classpath. Source java-test-args.sh before
+# running sbt to configure Gluten without replacing Delta's test fixtures.
#
# Usage:
-# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar>
<gluten_repo_root>
+# setup-delta.sh <delta_ref> <delta_dir> <gluten_bundle_jar>
#
# Arguments:
# delta_ref - git ref (tag/branch/sha) to check out (e.g. v4.4.1)
# delta_dir - destination directory for the Delta clone
# gluten_bundle_jar - path to the gluten-velox-bundle fat jar
-# gluten_repo_root - path to the Gluten repository root (used to locate
-#
backends-velox/src-delta4/.../DeltaSQLCommandTest.scala)
#
set -euo pipefail
-if [ "$#" -ne 4 ]; then
- echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar>
<gluten_repo_root>" >&2
+if [ "$#" -ne 3 ]; then
+ echo "Usage: $0 <delta_ref> <delta_dir> <gluten_bundle_jar>" >&2
exit 1
fi
DELTA_REF="$1"
DELTA_DIR="$2"
GLUTEN_BUNDLE_JAR="$3"
-GLUTEN_ROOT="$4"
if [ ! -f "$GLUTEN_BUNDLE_JAR" ]; then
echo "Gluten bundle jar not found: $GLUTEN_BUNDLE_JAR" >&2
exit 1
fi
-# Reuse the existing DeltaSQLCommandTest from Gluten's backends-velox module
-# rather than maintaining a separate copy. This file is compiled as part of the
-# unified `spark` project's Test scope, which has the Gluten bundle on its
-# classpath (via spark-unified/lib/), so the typed GlutenConfig /
VeloxDeltaConfig
-# imports resolve correctly.
-PATCH_SOURCE="$GLUTEN_ROOT/backends-velox/src-delta4/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
-if [ ! -f "$PATCH_SOURCE" ]; then
- echo "Gluten DeltaSQLCommandTest not found: $PATCH_SOURCE" >&2
- exit 1
-fi
-
echo "::group::Cloning delta-io/delta @ ${DELTA_REF}"
# init + shallow fetch resolves a tag, branch OR commit SHA in a single path
# (`git clone --branch` rejects SHAs). Avoids a full-clone fallback and the
@@ -81,7 +68,7 @@ echo "::endgroup::"
echo "::group::Injecting Gluten bundle jar onto the spark project's TEST
classpath"
# The Gluten bundle jar must be on the spark project's TEST runtime classpath
-# (so DeltaSQLCommandTest can load org.apache.gluten.GlutenPlugin by name) but
+# (so Spark can load org.apache.gluten.GlutenPlugin by name) but
# NOT on the COMPILE classpath of `sparkV1`, which is the project that holds
# Delta's main sources. The bundle's transitive contents include extra symbols
# under `org.apache.spark.sql` that collide with Delta's main sources -- e.g.
@@ -112,43 +99,5 @@ cp "$GLUTEN_BUNDLE_JAR"
"$SPARK_UNIFIED_LIB/gluten-velox-bundle.jar"
ls -lh "$SPARK_UNIFIED_LIB"
echo "::endgroup::"
-echo "::group::Patching DeltaSQLCommandTest to enable Gluten plugin"
-TARGET="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
-if [ ! -f "$TARGET" ]; then
- echo "Expected file not found in Delta clone: $TARGET" >&2
- echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
- exit 1
-fi
-cp "$PATCH_SOURCE" "$TARGET"
-echo "Patched $TARGET"
-echo "--- diff vs. upstream ---"
-git -C "$DELTA_DIR" --no-pager diff --
"spark/src/test/scala/org/apache/spark/sql/delta/test/DeltaSQLCommandTest.scala"
|| true
-echo "::endgroup::"
-
PATCH_SCRIPT="$(cd "$(dirname "${BASH_SOURCE[0]}")" &&
pwd)/apply-delta-test-patches.sh"
bash "$PATCH_SCRIPT" "$DELTA_REF" "$DELTA_DIR"
-
-echo "::group::Disabling Delta scalastyle HeaderMatchesChecker"
-# Our reused DeltaSQLCommandTest carries Gluten's ASF-only license header,
which
-# does not match Delta's HeaderMatchesChecker regex (the regex expects either a
-# Delta copyright block, or the ASF header followed by a Spark-modifications
-# block and the Delta copyright block). HeaderMatchesChecker is a file-level
-# checker that does NOT honor `// scalastyle:off` directives, so we instead
-# disable it globally in Delta's shared scalastyle-config.xml. The config is
-# applied via `ThisBuild / scalastyleConfig` in project/Checkstyle.scala, so a
-# single edit covers every sbt sub-project.
-SCALASTYLE_CONFIG="$DELTA_DIR/scalastyle-config.xml"
-if [ ! -f "$SCALASTYLE_CONFIG" ]; then
- echo "Expected scalastyle config not found: $SCALASTYLE_CONFIG" >&2
- exit 1
-fi
-sed -i \
- 's|<check level="error" class="org.scalastyle.file.HeaderMatchesChecker"
enabled="true">|<check level="error"
class="org.scalastyle.file.HeaderMatchesChecker" enabled="false">|' \
- "$SCALASTYLE_CONFIG"
-if ! grep -q '<check level="error"
class="org.scalastyle.file.HeaderMatchesChecker" enabled="false">'
"$SCALASTYLE_CONFIG"; then
- echo "Failed to disable HeaderMatchesChecker in $SCALASTYLE_CONFIG" >&2
- grep -n 'HeaderMatchesChecker' "$SCALASTYLE_CONFIG" >&2 || true
- exit 1
-fi
-echo "Disabled HeaderMatchesChecker in $SCALASTYLE_CONFIG"
-echo "::endgroup::"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
