This is an automated email from the ASF dual-hosted git repository.
philo-he pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 5e505c799f [GLUTEN-12753][CI] Match Delta CDF filter assertions to
Gluten plans (#13129)
5e505c799f is described below
commit 5e505c799fd011c29302e52317bd04a224fa0803
Author: Felipe Pessoto <[email protected]>
AuthorDate: Wed Oct 28 11:09:08 2026 -0700
[GLUTEN-12753][CI] Match Delta CDF filter assertions to Gluten plans
(#13129)
---
.github/workflows/util/delta-spark-ut/README.md | 42 ++++++-
.../delta-spark-ut/apply-delta-test-patches.sh | 121 ++++-----------------
.../util/delta-spark-ut/known-failures.txt | 13 ---
.../patches/cdf-pushed-filter-assertions.patch | 46 ++++++++
.../patches/dv-2b-row-fail-fast.patch | 34 ++++++
.../patches/dv-predicate-pushdown-row-groups.patch | 35 ++++++
.../patches/parquet-fixture-row-groups.patch | 27 +++++
7 files changed, 206 insertions(+), 112 deletions(-)
diff --git a/.github/workflows/util/delta-spark-ut/README.md
b/.github/workflows/util/delta-spark-ut/README.md
index 4f405fd594..78c4bb4d1a 100644
--- a/.github/workflows/util/delta-spark-ut/README.md
+++ b/.github/workflows/util/delta-spark-ut/README.md
@@ -43,7 +43,47 @@ starts failing** (a regression).
| `run-delta-tests.sh` | The shard step's body: runs `sbt spark/test` (tuned
JVM/heap flags) under a hang watchdog, prints memory forensics, then gates the
results against the baseline via `compare-test-results.py`. |
| `java-test-args.sh` | Shared JVM flags (`--add-opens` + Netty property)
needed to run the suite on JDK 17 with the Gluten bundle. Sourced by
`run-delta-tests.sh` and by local runs. |
| `setup-delta.sh` | Clones Delta, drops in the Gluten bundle, and patches
`DeltaSQLCommandTest`. |
-| `apply-delta-test-patches.sh` | Applies temporary Delta test workarounds:
upstream scan fixes, deterministic Parquet row groups, and 2B-row DV fail-fast
patches. Called by `setup-delta.sh` with `<delta_ref> <delta_dir>`. |
+| `apply-delta-test-patches.sh` | Cherry-picks upstream scan fixes, applies
the ordered local patch list, and verifies the resulting test sources. Called
by `setup-delta.sh` with `<delta_ref> <delta_dir>`. |
+| `patches/*.patch` | Separate patches for CDF plan assertions, the two
Parquet row-group fixtures, and 2B-row DV fail-fast behavior. Each patch
documents its rationale and removal condition. |
+
+## Temporary Delta test patches
+
+Local workarounds live in `patches/`, one file per purpose. The patcher applies
+an explicit ordered list with `git apply` after the upstream cherry-picks,
+then verifies the expected row-count-scope and fail-fast counts.
+Patch paths are resolved relative to the script, independent of the caller's
+working directory. Missing patches, incompatible source context, or unexpected
+counts fail setup explicitly. Run through `setup-delta.sh` to reset the Delta
+checkout before applying the patches again.
+
+The `dv-2b-row-fail-fast.patch` keeps the two memory-hog DV tests failing
+immediately rather than skipping them. Their known-failure entries remain
+enforced until the underlying native-memory issue is fixed.
+
+## Temporary CDF plan-assertion workaround
+
+Gluten's native file scans currently display `PushedFilters` without Spark's
+handled-filter `*` marker
+([#12753](https://github.com/apache/gluten/issues/12753)).
+`patches/cdf-pushed-filter-assertions.patch` adjusts three shared assertions in
+the CI-created Delta checkout, covering 13 CDF test cases, to require Gluten's
+exact display:
+
+```text
+PushedFilters: [IsNotNull(id), LessThan(id,5)]
+PushedFilters: [IsNotNull(`id with space`), LessThan(`id with space`,5)]
+```
+
+The column-mapping cases also need the backticks around the special-character
+identifier; removing only `*` would not make those assertions match. Predicate
+contents, selected-column assertions, and result checks remain unchanged.
+The patch is applied after the upstream cherry-picks; `git apply` fails setup
+if the required files or source context are missing.
+
+This is a test-only accommodation, not a fix for Gluten's scan display. Remove
+it when the display gap is fixed or Delta's assertions no longer require the
+workaround. The affected cases are no longer in `known-failures.txt`, so their
+failures remain enforced by the gate.
## How the gate works
diff --git a/.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh
b/.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh
index a1956bcc3f..b1bc85c085 100755
--- a/.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh
+++ b/.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh
@@ -34,6 +34,7 @@ fi
DELTA_REF="$1"
DELTA_DIR="$2"
+PATCH_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/patches"
# Delta's tests collect file-source scans by matching the concrete
# `FileSourceScanExec` case class; Gluten offloads the scan to
@@ -82,43 +83,31 @@ cherry_pick_delta_fix
46bd45d57eadd7e528002a0ae7bd36ce5a456eca "#7104 (ScanRepor
cherry_pick_delta_fix 959e00e15f41f56afc1c9bb95d160c55c6dc7068 "#7105 (9 more
test suites)"
echo "::endgroup::"
-echo "::group::Capping DeltaParquetFileFormat fixture row groups by row count"
-# DeltaParquetFileFormatSuite generates one 20,000-row Parquet file and sets a
-# 50 KiB block size to ensure that it contains multiple row groups. Velox sizes
-# row groups by buffered bytes after writing each input batch. Because this
-# fixture arrives in one batch, lowering the byte threshold cannot split it.
-# Scope Gluten's native row-count limit around the fixture write so Arrow
splits
-# the 20,000 rows deterministically while keeping the native write path
enabled.
+echo "::group::Applying temporary Delta test patches"
+# Keep local patches after the cherry-picks, which require an unmodified
worktree
+# for the files they touch. Patch paths are relative to this script, not Delta.
+for patch in \
+ cdf-pushed-filter-assertions.patch \
+ parquet-fixture-row-groups.patch \
+ dv-predicate-pushdown-row-groups.patch \
+ dv-2b-row-fail-fast.patch; do
+ echo "Applying $patch"
+ if ! git -C "$DELTA_DIR" apply "$PATCH_DIR/$patch"; then
+ echo "ERROR: Delta test patch '$patch' did not apply." >&2
+ echo "The patch expects Delta v4.4.1 test sources;" \
+ "ref '${DELTA_REF}' must remain source-compatible." >&2
+ exit 1
+ fi
+done
+echo "::endgroup::"
+
+echo "::group::Checking DeltaParquetFileFormat fixture row groups"
DPFFS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala"
if [ ! -f "$DPFFS" ]; then
echo "Expected file not found in Delta clone: $DPFFS" >&2
echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
>&2
exit 1
fi
-if ! sed 's/^__BLANK_CONTEXT__$/ /' <<'PATCH' | git -C "$DELTA_DIR" apply -
-diff --git
a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
----
a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
-+++
b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
-@@ -68,9 +68,11 @@ trait DeltaParquetFileFormatSuiteBase
- protected def generateData(tablePath: String): Unit = {
- // This is to generate a Parquet file with two row groups
- hadoopConf().set("parquet.block.size", (1024 * 50).toString)
-__BLANK_CONTEXT__
- // Keep the number of partitions to 1 to generate a single Parquet data
file
- val df = Seq.range(0, 20000).toDF().repartition(1)
-- df.write.format("delta").mode("append").save(tablePath)
-+ withSQLConf("spark.gluten.sql.native.parquet.write.blockRows" -> "10000")
{
-+ df.write.format("delta").mode("append").save(tablePath)
-+ }
-__BLANK_CONTEXT__
- // Set DFS block size to be less than Parquet rowgroup size, to allow
-PATCH
-then
- echo "ERROR: DeltaParquetFileFormat fixture patch did not apply." >&2
- echo "The patch expects the Delta v4.4.1 generateData fixture shape;" \
- "ref '${DELTA_REF}' must remain source-compatible." >&2
- exit 1
-fi
ROW_CAP_SCOPES=$(
grep -Fxc \
' withSQLConf("spark.gluten.sql.native.parquet.write.blockRows" ->
"10000") {' \
@@ -135,51 +124,13 @@ git -C "$DELTA_DIR" --no-pager diff -- \
"spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala"
|| true
echo "::endgroup::"
-echo "::group::Capping predicate-pushdown DV fixture row groups by row count"
-# DeletionVectorsWithPredicatePushdownSuite writes one 1,000,000-row Parquet
-# file and expects its 2 MiB block size to produce two row groups. The rows can
-# arrive at Velox in one Arrow batch, so the native writer cannot evaluate its
-# buffered-byte flush threshold until the whole batch is already in one group.
-# Cap this fixture at 500,000 rows per group so Arrow splits the batch while
-# retaining the native write path and the existing Hadoop block-size setting.
+echo "::group::Checking predicate-pushdown DV fixture row groups"
DV_SUITE="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala"
if [ ! -f "$DV_SUITE" ]; then
echo "Expected file not found in Delta clone: $DV_SUITE" >&2
echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
>&2
exit 1
fi
-if ! sed 's/^__BLANK_CONTEXT__$/ /' <<'PATCH' | git -C "$DELTA_DIR" apply -
-diff --git
a/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala
b/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala
----
a/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala
-+++
b/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala
-@@ -913,12 +913,14 @@ class DeletionVectorsWithPredicatePushdownSuite extends
DeletionVectorsSuite {
- super.beforeAll()
-__BLANK_CONTEXT__
- // 2MB rowgroups.
- hadoopConf().set("parquet.block.size", (2 * 1024 * 1024).toString)
-__BLANK_CONTEXT__
-- spark.range(0, multiRowgroupTableRowsNum, 1, 1).toDF("id")
-- .write
-- .option(DeltaConfigs.ENABLE_DELETION_VECTORS_CREATION.key,
true.toString)
-- .format("delta")
-- .saveAsTable(multiRowgroupTable)
-+ withSQLConf("spark.gluten.sql.native.parquet.write.blockRows" ->
"500000") {
-+ spark.range(0, multiRowgroupTableRowsNum, 1, 1).toDF("id")
-+ .write
-+ .option(DeltaConfigs.ENABLE_DELETION_VECTORS_CREATION.key,
true.toString)
-+ .format("delta")
-+ .saveAsTable(multiRowgroupTable)
-+ }
-__BLANK_CONTEXT__
- val deltaLog = DeltaLog.forTable(spark,
TableIdentifier(multiRowgroupTable))
- val files = deltaLog.update().allFiles.collect()
-PATCH
-then
- echo "ERROR: predicate-pushdown DV fixture patch did not apply." >&2
- echo "The patch expects the Delta v4.4.1 beforeAll fixture shape;" \
- "ref '${DELTA_REF}' must remain source-compatible." >&2
- exit 1
-fi
DV_ROW_CAP_SCOPES=$(
grep -Fxc \
' withSQLConf("spark.gluten.sql.native.parquet.write.blockRows" ->
"500000") {' \
@@ -196,43 +147,17 @@ git -C "$DELTA_DIR" --no-pager diff -- \
"spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala"
|| true
echo "::endgroup::"
-echo "::group::Force-failing memory-hog DeletionVectorsSuite 2B-row tests"
-# Two DeletionVectorsSuite tests read from / delete from a 2-billion-row table.
-# Under the Gluten Velox bundle they balloon the forked test JVM to ~13G of
-# NATIVE memory (row-index materialization) and the kernel/cgroup OOM-kills it.
-# The dead fork then wedges sbt, hanging the whole shard until the workflow's
-# hang-watchdog dumps threads and kills it (~16 min wasted, and every suite
-# QUEUED AFTER it in that fork is skipped) -- see delta_spark_ut.yml.
-#
-# Rather than silently `ignore` these (easy to forget), we make them FAIL FAST
-# with a clear message: the gap stays visible in the test reports / baseline
-# until the native memory blow-up is fixed, at which point this patch should be
-# removed. NOTE: making the suite complete also un-skips the rest of the
shard's
-# suite queue, so the known-failures baseline must be refreshed after this.
-#
-# ORDER MATTERS: keep this sed AFTER the cherry-picks above. #7105 also edits
-# DeletionVectorsSuite.scala, and git cherry-pick aborts (exit 128) when the
work
-# tree has uncommitted edits to a file it touches. It also relies on the clone
-# step's `checkout -f`: the sed appends after the declaration line, so without
-# that per-run reset a re-run injects duplicate `fail` lines and trips the
-# INJECTED != 2 check below.
+echo "::group::Checking DeletionVectorsSuite 2B-row fail-fast guards"
DVS="$DELTA_DIR/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala"
if [ ! -f "$DVS" ]; then
echo "Expected file not found in Delta clone: $DVS" >&2
echo "The Delta directory layout for ref '${DELTA_REF}' may have changed."
>&2
exit 1
fi
-# Inject `fail(...)` as the first statement of each test body (the line ending
-# in `) {`). Delta sets no -Xfatal-warnings / dead-code warning, so the now-
-# unreachable original body compiles fine. Keep each injected line <100 chars:
-# Delta's scalastyle enforces a 100-char line length on test sources. The full
-# rationale lives in this comment, so the in-test message stays terse.
-sed -i 's#huge table: read from tables of 2B rows with existing DV of many
zeros") {#&\n fail("[Gluten CI] Force-failed: 2B-row DV read OOMs the test
JVM; see setup-delta.sh")#' "$DVS"
-sed -i 's#number of rows from tables of 2B rows with DVs") {#&\n
fail("[Gluten CI] Force-failed: 2B-row DV delete OOMs the test JVM; see
setup-delta.sh")#' "$DVS"
INJECTED=$(grep -c "Gluten CI] Force-failed" "$DVS" || true)
if [ "$INJECTED" -ne 2 ]; then
echo "ERROR: expected to force-fail 2 DeletionVectorsSuite tests but
injected ${INJECTED}." >&2
- echo "Their test names likely changed in Delta ref '${DELTA_REF}'; update
apply-delta-test-patches.sh." >&2
+ echo "Their test names likely changed in Delta ref '${DELTA_REF}'; update
dv-2b-row-fail-fast.patch." >&2
exit 1
fi
echo "Force-failed 2 DeletionVectorsSuite 2B-row tests (read + delete)."
diff --git a/.github/workflows/util/delta-spark-ut/known-failures.txt
b/.github/workflows/util/delta-spark-ut/known-failures.txt
index 827e95356b..6bb839c88f 100644
--- a/.github/workflows/util/delta-spark-ut/known-failures.txt
+++ b/.github/workflows/util/delta-spark-ut/known-failures.txt
@@ -68,19 +68,6 @@ org.apache.spark.sql.delta.ConvertToDeltaScalaSuite#CONVERT
keeps NaN rows for f
org.apache.spark.sql.delta.ConvertToDeltaScalaSuite#external tables use
correct path scheme
org.apache.spark.sql.delta.DeleteMetricsSuite#delete-metrics: delete one row
per file - Partitioned = false, cdfEnabled = false
org.apache.spark.sql.delta.DeleteMetricsSuite#delete-metrics: delete one row
per file - Partitioned = false, cdfEnabled = true
-org.apache.spark.sql.delta.DeltaCDCIdColumnMappingSuite#filters with special
characters in name should be pushed down - column mapping id mode
-org.apache.spark.sql.delta.DeltaCDCNameColumnMappingSuite#filters with special
characters in name should be pushed down - column mapping name mode
-org.apache.spark.sql.delta.DeltaCDCSQLIdColumnMappingSuite#filters with
special characters in name should be pushed down - column mapping id mode
-org.apache.spark.sql.delta.DeltaCDCSQLNameColumnMappingSuite#filters with
special characters in name should be pushed down - column mapping name mode
-org.apache.spark.sql.delta.DeltaCDCSQLSuite#select individual column should
push down filters
-org.apache.spark.sql.delta.DeltaCDCSQLWithCatalogOwnedBatch100Suite#select
individual column should push down filters
-org.apache.spark.sql.delta.DeltaCDCSQLWithCatalogOwnedBatch1Suite#select
individual column should push down filters
-org.apache.spark.sql.delta.DeltaCDCSQLWithCatalogOwnedBatch2Suite#select
individual column should push down filters
-org.apache.spark.sql.delta.DeltaCDCScalaSuite#filters should be pushed down
-org.apache.spark.sql.delta.DeltaCDCScalaWithCatalogOwnedBatch100Suite#filters
should be pushed down
-org.apache.spark.sql.delta.DeltaCDCScalaWithCatalogOwnedBatch1Suite#filters
should be pushed down
-org.apache.spark.sql.delta.DeltaCDCScalaWithCatalogOwnedBatch2Suite#filters
should be pushed down
-org.apache.spark.sql.delta.DeltaCDCScalaWithDeletionVectorsSuite#filters
should be pushed down
org.apache.spark.sql.delta.DeltaCDCStreamDeletionVectorSuite#cdc streams with
noop merge
org.apache.spark.sql.delta.DeltaCDCStreamSuite#cdc streams with noop merge
org.apache.spark.sql.delta.DeltaCDCStreamWithCatalogManagedBatch100Suite#cdc
streams with noop merge
diff --git
a/.github/workflows/util/delta-spark-ut/patches/cdf-pushed-filter-assertions.patch
b/.github/workflows/util/delta-spark-ut/patches/cdf-pushed-filter-assertions.patch
new file mode 100644
index 0000000000..0b62cd50f6
--- /dev/null
+++
b/.github/workflows/util/delta-spark-ut/patches/cdf-pushed-filter-assertions.patch
@@ -0,0 +1,46 @@
+Subject: [PATCH] Match Delta CDF pushed-filter assertions to Gluten
+
+Gluten's file scans omit Spark's handled-filter '*' marker
+(apache/gluten#12753). They also quote special-character column names,
+unlike Delta's vanilla CDF scan.
+
+Adjust only these three shared assertions, covering 13 concrete test cases;
+keep their predicate, column-pruning, and result checks intact. Remove this
+workaround when #12753 is fixed or Delta no longer requires the starred form.
+
+diff --git
a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaCDCSuite.scala
b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaCDCSuite.scala
+--- a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaCDCSuite.scala
++++ b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaCDCSuite.scala
+@@ -1045,7 +1045,7 @@ class DeltaCDCScalaSuite extends DeltaCDCSuiteBase {
+ .withColumn("_change_type", lit("insert")))
+ }
+ assert(plans.map(_.executedPlan).toString
+- .contains("PushedFilters: [*IsNotNull(id), *LessThan(id,5)]"))
++ .contains("PushedFilters: [IsNotNull(id), LessThan(id,5)]"))
+ }
+ }
+
+diff --git
a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaCDCSQLSuite.scala
b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaCDCSQLSuite.scala
+--- a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaCDCSQLSuite.scala
++++ b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaCDCSQLSuite.scala
+@@ -145,7 +145,7 @@ class DeltaCDCSQLSuite extends DeltaCDCSuiteBase with
DeltaColumnMappingTestUtils {
+ .withColumn("_change_type", lit("insert")))
+ }
+ assert(plans.map(_.executedPlan).toString
+- .contains("PushedFilters: [*IsNotNull(id), *LessThan(id,5)]"))
++ .contains("PushedFilters: [IsNotNull(id), LessThan(id,5)]"))
+ }
+ }
+
+diff --git
a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaCDCColumnMappingSuite.scala
b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaCDCColumnMappingSuite.scala
+---
a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaCDCColumnMappingSuite.scala
++++
b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaCDCColumnMappingSuite.scala
+@@ -668,7 +668,7 @@ trait DeltaCDCColumnMappingSuiteBase extends
DeltaColumnMappingSelectedTestMixin {
+ .withColumn("_change_type", lit("insert")))
+ }
+ assert(plans.map(_.executedPlan).toString
+- .contains("PushedFilters: [*IsNotNull(id with space), *LessThan(id
with space,5)]"))
++ .contains("PushedFilters: [IsNotNull(`id with space`), LessThan(`id
with space`,5)]"))
+ }
+ }
+ }
diff --git
a/.github/workflows/util/delta-spark-ut/patches/dv-2b-row-fail-fast.patch
b/.github/workflows/util/delta-spark-ut/patches/dv-2b-row-fail-fast.patch
new file mode 100644
index 0000000000..badad1c69b
--- /dev/null
+++ b/.github/workflows/util/delta-spark-ut/patches/dv-2b-row-fail-fast.patch
@@ -0,0 +1,34 @@
+Subject: [PATCH] Fail fast in memory-hog DeletionVectorsSuite 2B-row tests
+
+These tests read from or delete from a 2-billion-row table. With Gluten,
+row-index materialization balloons the forked test JVM to about 13 GiB of
+native memory and the kernel/cgroup OOM-kills it. The dead fork wedges sbt
+until the workflow watchdog intervenes, skipping suites queued after it.
+
+Fail fast rather than silently ignoring these tests so the gap stays visible
+in test reports and the known-failures baseline. Remove this patch and refresh
+the baseline when the native memory issue is fixed.
+
+Keep the fail calls first in each test body and below Delta's 100-character
+line limit. Delta does not enable fatal dead-code warnings, so the original
+test bodies can remain unchanged.
+
+diff --git
a/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala
b/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala
+---
a/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala
++++
b/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala
+@@ -664,6 +664,7 @@ class DeletionVectorsSuite extends QueryTest
+ }
+
+ test("huge table: read from tables of 2B rows with existing DV of many
zeros") {
++ fail("[Gluten CI] Force-failed: 2B-row DV read OOMs the test JVM; see
setup-delta.sh")
+ val canonicalTable5Path = new File(table5Path).getCanonicalPath
+
+ val predicatePushDownEnabled =
+@@ -784,6 +785,7 @@ class DeletionVectorsSuite extends QueryTest
+ for (deleteSpec <- Seq(deleteUsingDvSmallScale)) {
+ test(
+ s"huge table: delete a ${deleteSpec.scale} number of rows from tables
of 2B rows with DVs") {
++ fail("[Gluten CI] Force-failed: 2B-row DV delete OOMs the test JVM; see
setup-delta.sh")
+ val predicatePushDownEnabled =
+ spark.conf.get(DeltaSQLConf.DELETION_VECTORS_USE_METADATA_ROW_INDEX)
+ withTempDir { dir =>
diff --git
a/.github/workflows/util/delta-spark-ut/patches/dv-predicate-pushdown-row-groups.patch
b/.github/workflows/util/delta-spark-ut/patches/dv-predicate-pushdown-row-groups.patch
new file mode 100644
index 0000000000..099155c943
--- /dev/null
+++
b/.github/workflows/util/delta-spark-ut/patches/dv-predicate-pushdown-row-groups.patch
@@ -0,0 +1,35 @@
+Subject: [PATCH] Cap predicate-pushdown DV fixture row groups by row count
+
+DeletionVectorsWithPredicatePushdownSuite writes one 1,000,000-row Parquet
+file and expects its 2 MiB block size to produce two row groups. The rows can
+arrive at Velox in one Arrow batch, so the native writer cannot evaluate its
+buffered-byte flush threshold until the whole batch is already in one group.
+
+Cap this fixture at 500,000 rows per group so Arrow splits the batch while
+retaining the native write path and the existing Hadoop block-size setting.
+Remove this workaround when the native writer honors the fixture's byte limit.
+
+diff --git
a/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala
b/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala
+---
a/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala
++++
b/spark/src/test/scala/org/apache/spark/sql/delta/deletionvectors/DeletionVectorsSuite.scala
+@@ -913,12 +913,14 @@ class DeletionVectorsWithPredicatePushdownSuite extends
DeletionVectorsSuite {
+ super.beforeAll()
+
+ // 2MB rowgroups.
+ hadoopConf().set("parquet.block.size", (2 * 1024 * 1024).toString)
+
+- spark.range(0, multiRowgroupTableRowsNum, 1, 1).toDF("id")
+- .write
+- .option(DeltaConfigs.ENABLE_DELETION_VECTORS_CREATION.key,
true.toString)
+- .format("delta")
+- .saveAsTable(multiRowgroupTable)
++ withSQLConf("spark.gluten.sql.native.parquet.write.blockRows" ->
"500000") {
++ spark.range(0, multiRowgroupTableRowsNum, 1, 1).toDF("id")
++ .write
++ .option(DeltaConfigs.ENABLE_DELETION_VECTORS_CREATION.key,
true.toString)
++ .format("delta")
++ .saveAsTable(multiRowgroupTable)
++ }
+
+ val deltaLog = DeltaLog.forTable(spark,
TableIdentifier(multiRowgroupTable))
+ val files = deltaLog.update().allFiles.collect()
diff --git
a/.github/workflows/util/delta-spark-ut/patches/parquet-fixture-row-groups.patch
b/.github/workflows/util/delta-spark-ut/patches/parquet-fixture-row-groups.patch
new file mode 100644
index 0000000000..55ca96322f
--- /dev/null
+++
b/.github/workflows/util/delta-spark-ut/patches/parquet-fixture-row-groups.patch
@@ -0,0 +1,27 @@
+Subject: [PATCH] Cap DeltaParquetFileFormat fixture row groups by row count
+
+DeltaParquetFileFormatSuite generates one 20,000-row Parquet file and sets a
+50 KiB block size to ensure that it contains multiple row groups. Velox sizes
+row groups by buffered bytes after writing each input batch. Because this
+fixture arrives in one batch, lowering the byte threshold cannot split it.
+
+Scope Gluten's native row-count limit around the fixture write so Arrow splits
+the 20,000 rows deterministically while keeping the native write path enabled.
+Remove this workaround when the native writer honors the fixture's byte limit.
+
+diff --git
a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
+---
a/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
++++
b/spark/src/test/scala/org/apache/spark/sql/delta/DeltaParquetFileFormatSuite.scala
+@@ -68,9 +68,11 @@ trait DeltaParquetFileFormatSuiteBase
+ protected def generateData(tablePath: String): Unit = {
+ // This is to generate a Parquet file with two row groups
+ hadoopConf().set("parquet.block.size", (1024 * 50).toString)
+
+ // Keep the number of partitions to 1 to generate a single Parquet data
file
+ val df = Seq.range(0, 20000).toDF().repartition(1)
+- df.write.format("delta").mode("append").save(tablePath)
++ withSQLConf("spark.gluten.sql.native.parquet.write.blockRows" -> "10000")
{
++ df.write.format("delta").mode("append").save(tablePath)
++ }
+
+ // Set DFS block size to be less than Parquet rowgroup size, to allow
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
