This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new caa83df7fb [GLUTEN-12857][TEST] Fix V1 DPP suites extending the V2
base (#13620)
caa83df7fb is described below
commit caa83df7fb6569827e1a70c7a85a022e7b08d983
Author: Shivadarshan Devadiga <[email protected]>
AuthorDate: Thu Nov 12 18:11:38 2026 +0530
[GLUTEN-12857][TEST] Fix V1 DPP suites extending the V2 base (#13620)
## What changes are proposed in this pull request?
Fixes #12857
Three DPP suites carry `V1` in their name but extend
`GlutenDynamicPartitionPruningV2SuiteAEOff`, so they re-run the V2 scenario
instead of covering the V1 DataSource path:
- `GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan`
- `GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject`
- `GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOffDisableProject`
`V1SuiteAEOffDisableScan` sets `COLUMNAR_FILESCAN_ENABLED=false`, which has
no effect on the V2 batch scan. The two `V1...DisableProject` suites were
identical to their `V2...` counterparts. The AQE-on variant,
`V1SuiteAEOnDisableScan`, already extends the V1 base. This PR changes the
three suites to extend `GlutenDynamicPartitionPruningV1SuiteAEOff` in all five
modules (`gluten-ut/spark34`, `spark35`, `spark40`, `spark41`, `spark42`).
Running them on the V1 base exposed one test that never ran with the file
scan disabled: the Gluten copy of `Subquery reuse across the whole plan` in
`spark34`/`spark35`. It expects 3 subqueries and 2 reuses, because Gluten's
native scan also pushes down the subquery filter
(`FilterHandler#applyFilterPushdownToScan`). With the scan falling back to
vanilla Spark, the plan has vanilla Spark's 2 and 1, so the test failed. The
test now takes its expectation from `COLUMNAR_FILESCAN_ENABLED [...]
ClickHouse excludes this Gluten test on `V1SuiteAEOff` in spark35.
`V1SuiteAEOffDisableScan` now inherits it, so the same exclusion is added to
that entry. The other backends and versions inherit nothing new that they
exclude on the parent.
## How was this patch tested?
- Ran `GlutenDynamicPartitionPruningV1SuiteAEOff`,
`V1SuiteAEOffDisableScan`, `V1SuiteAEOffWSCGOnDisableProject` and
`V1SuiteAEOffWSCGOffDisableProject` locally on the Velox backend, on each Spark
version, with ANSI off as in the regular CI:
| Spark | Tests passed | Failed |
|---|---|---|
| 3.4 | 152 | 0 |
| 3.5 | 152 | 0 |
| 4.0 | 152 | 0 |
| 4.1 | 152 | 0 |
| 4.2 | 152 | 0 |
- On 3.4/3.5 the base suite takes the 3/2 branch of `Subquery reuse across
the whole plan` and `V1SuiteAEOffDisableScan` takes the 2/1 branch. Before the
test change, `V1SuiteAEOffDisableScan` failed it with `ArrayBuffer(...) had
size 2 instead of expected size 3`.
- `spotless:check` passes for `gluten-ut/spark34` and `gluten-ut/spark35`.
- The ClickHouse and Bolt backends were not run locally; CI covers them.
The three `Disable*` suites are the first runs of these V1 paths, so if a
backend's CI flags one of them, the inherited V1 tests are the place to look.
## Was this patch authored or co-authored using generative AI tooling?
Co-authored by: Claude Sonnet 4.5
---
.../sql/GlutenDynamicPartitionPruningSuite.scala | 23 ++++++++++++++++------
.../utils/clickhouse/ClickHouseTestSettings.scala | 1 +
.../sql/GlutenDynamicPartitionPruningSuite.scala | 23 ++++++++++++++++------
.../sql/GlutenDynamicPartitionPruningSuite.scala | 6 +++---
.../sql/GlutenDynamicPartitionPruningSuite.scala | 6 +++---
.../sql/GlutenDynamicPartitionPruningSuite.scala | 6 +++---
6 files changed, 44 insertions(+), 21 deletions(-)
diff --git
a/gluten-ut/spark34/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
b/gluten-ut/spark34/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
index 3b2be3c72e..5d7c18d082 100644
---
a/gluten-ut/spark34/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
+++
b/gluten-ut/spark34/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
@@ -722,12 +722,23 @@ class GlutenDynamicPartitionPruningV1SuiteAEOff
case rs: ReusedSubqueryExec => rs.child.id
}
- // By default Gluten pushes more filters than vanilla Spark.
+ // By default Gluten pushes more filters than vanilla Spark. When the
file scan is offloaded
+ // (COLUMNAR_FILESCAN_ENABLED), the native scan also takes the
subquery filter, which adds
+ // one subquery and one reuse. With the file scan disabled, the plan
has vanilla Spark's
+ // counts.
//
// See also
org.apache.gluten.execution.FilterHandler#applyFilterPushdownToScan
// See also DynamicPartitionPruningSuite.scala:1362
- assert(subqueryIds.size == 3, "Whole plan subquery reusing not working
correctly")
- assert(reusedSubqueryIds.size == 2, "Whole plan subquery reusing not
working correctly")
+ val scanOffloaded = GlutenConfig.get.enableColumnarFileScan
+ val nativeScan = plan.collectWithSubqueries { case s:
FileSourceScanExecTransformer => s }
+ assert(nativeScan.nonEmpty == scanOffloaded, "Unexpected file scan
offloading")
+ val (expectedSubqueries, expectedReused) = if (scanOffloaded) (3, 2)
else (2, 1)
+ assert(
+ subqueryIds.size == expectedSubqueries,
+ "Whole plan subquery reusing not working correctly")
+ assert(
+ reusedSubqueryIds.size == expectedReused,
+ "Whole plan subquery reusing not working correctly")
assert(
reusedSubqueryIds.forall(subqueryIds.contains(_)),
"ReusedSubqueryExec should reuse an existing subquery")
@@ -806,7 +817,7 @@ class GlutenDynamicPartitionPruningV1SuiteAEOnDisableScan
// Same as above except AQE is off.
class GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan
- extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+ extends GlutenDynamicPartitionPruningV1SuiteAEOff {
override def sparkConf: SparkConf = {
super.sparkConf.set(GlutenConfig.COLUMNAR_FILESCAN_ENABLED.key, "false")
}
@@ -832,7 +843,7 @@ class GlutenDynamicPartitionPruningV2SuiteAEOffDisableScan
// Test DPP with project disabled by user for some reason, which can also mock
the situation
// that project is not transformable.
class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
- extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+ extends GlutenDynamicPartitionPruningV1SuiteAEOff {
override def sparkConf: SparkConf = {
super.sparkConf.set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
}
@@ -840,7 +851,7 @@ class
GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
// Same as above except whole-stage codegen is off.
class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOffDisableProject
- extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+ extends GlutenDynamicPartitionPruningV1SuiteAEOff {
override def sparkConf: SparkConf = {
super.sparkConf
.set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
diff --git
a/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/clickhouse/ClickHouseTestSettings.scala
b/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/clickhouse/ClickHouseTestSettings.scala
index 788a5c82ef..ca81410bb5 100644
---
a/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/clickhouse/ClickHouseTestSettings.scala
+++
b/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/clickhouse/ClickHouseTestSettings.scala
@@ -769,6 +769,7 @@ class ClickHouseTestSettings extends BackendTestSettings {
enableSuite[GlutenDynamicPartitionPruningV1SuiteAEOff]
.excludeGlutenTest("Subquery reuse across the whole plan")
enableSuite[GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan]
+ .excludeGlutenTest("Subquery reuse across the whole plan")
enableSuite[GlutenDynamicPartitionPruningV1SuiteAEOn]
enableSuite[GlutenDynamicPartitionPruningV1SuiteAEOnDisableScan]
enableSuite[GlutenDynamicPartitionPruningV2SuiteAEOff]
diff --git
a/gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
b/gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
index ab34d46f67..d6293f7ce6 100644
---
a/gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
+++
b/gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
@@ -619,12 +619,23 @@ class GlutenDynamicPartitionPruningV1SuiteAEOff
case rs: ReusedSubqueryExec => rs.child.id
}
- // By default Gluten pushes more filters than vanilla Spark.
+ // By default Gluten pushes more filters than vanilla Spark. When the
file scan is offloaded
+ // (COLUMNAR_FILESCAN_ENABLED), the native scan also takes the
subquery filter, which adds
+ // one subquery and one reuse. With the file scan disabled, the plan
has vanilla Spark's
+ // counts.
//
// See also
org.apache.gluten.execution.FilterHandler#applyFilterPushdownToScan
// See also DynamicPartitionPruningSuite.scala:1362
- assert(subqueryIds.size == 3, "Whole plan subquery reusing not working
correctly")
- assert(reusedSubqueryIds.size == 2, "Whole plan subquery reusing not
working correctly")
+ val scanOffloaded = GlutenConfig.get.enableColumnarFileScan
+ val nativeScan = plan.collectWithSubqueries { case s:
FileSourceScanExecTransformer => s }
+ assert(nativeScan.nonEmpty == scanOffloaded, "Unexpected file scan
offloading")
+ val (expectedSubqueries, expectedReused) = if (scanOffloaded) (3, 2)
else (2, 1)
+ assert(
+ subqueryIds.size == expectedSubqueries,
+ "Whole plan subquery reusing not working correctly")
+ assert(
+ reusedSubqueryIds.size == expectedReused,
+ "Whole plan subquery reusing not working correctly")
assert(
reusedSubqueryIds.forall(subqueryIds.contains(_)),
"ReusedSubqueryExec should reuse an existing subquery")
@@ -746,7 +757,7 @@ class GlutenDynamicPartitionPruningV1SuiteAEOnDisableScan
// Same as above except AQE is off.
class GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan
- extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+ extends GlutenDynamicPartitionPruningV1SuiteAEOff {
override def sparkConf: SparkConf = {
super.sparkConf.set(GlutenConfig.COLUMNAR_FILESCAN_ENABLED.key, "false")
}
@@ -772,7 +783,7 @@ class GlutenDynamicPartitionPruningV2SuiteAEOffDisableScan
// Test DPP with project disabled by user for some reason, which can also mock
the situation
// that project is not transformable.
class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
- extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+ extends GlutenDynamicPartitionPruningV1SuiteAEOff {
override def sparkConf: SparkConf = {
super.sparkConf.set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
}
@@ -780,7 +791,7 @@ class
GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
// Same as above except whole-stage codegen is off.
class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOffDisableProject
- extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+ extends GlutenDynamicPartitionPruningV1SuiteAEOff {
override def sparkConf: SparkConf = {
super.sparkConf
.set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
diff --git
a/gluten-ut/spark40/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
b/gluten-ut/spark40/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
index 9f4e18fb82..aee71ec88e 100644
---
a/gluten-ut/spark40/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
+++
b/gluten-ut/spark40/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
@@ -687,7 +687,7 @@ class GlutenDynamicPartitionPruningV1SuiteAEOnDisableScan
// Same as above except AQE is off.
class GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan
- extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+ extends GlutenDynamicPartitionPruningV1SuiteAEOff {
override def sparkConf: SparkConf = {
super.sparkConf.set(GlutenConfig.COLUMNAR_FILESCAN_ENABLED.key, "false")
}
@@ -713,7 +713,7 @@ class GlutenDynamicPartitionPruningV2SuiteAEOffDisableScan
// Test DPP with project disabled by user for some reason, which can also mock
the situation
// that project is not transformable.
class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
- extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+ extends GlutenDynamicPartitionPruningV1SuiteAEOff {
override def sparkConf: SparkConf = {
super.sparkConf.set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
}
@@ -721,7 +721,7 @@ class
GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
// Same as above except whole-stage codegen is off.
class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOffDisableProject
- extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+ extends GlutenDynamicPartitionPruningV1SuiteAEOff {
override def sparkConf: SparkConf = {
super.sparkConf
.set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
diff --git
a/gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
b/gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
index 2a1a2c55fe..0ac03f1b18 100644
---
a/gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
+++
b/gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
@@ -689,7 +689,7 @@ class GlutenDynamicPartitionPruningV1SuiteAEOnDisableScan
// Same as above except AQE is off.
class GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan
- extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+ extends GlutenDynamicPartitionPruningV1SuiteAEOff {
override def sparkConf: SparkConf = {
super.sparkConf.set(GlutenConfig.COLUMNAR_FILESCAN_ENABLED.key, "false")
}
@@ -715,7 +715,7 @@ class GlutenDynamicPartitionPruningV2SuiteAEOffDisableScan
// Test DPP with project disabled by user for some reason, which can also mock
the situation
// that project is not transformable.
class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
- extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+ extends GlutenDynamicPartitionPruningV1SuiteAEOff {
override def sparkConf: SparkConf = {
super.sparkConf.set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
}
@@ -723,7 +723,7 @@ class
GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
// Same as above except whole-stage codegen is off.
class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOffDisableProject
- extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+ extends GlutenDynamicPartitionPruningV1SuiteAEOff {
override def sparkConf: SparkConf = {
super.sparkConf
.set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
diff --git
a/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
b/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
index 84a9107852..418dde7138 100644
---
a/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
+++
b/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
@@ -689,7 +689,7 @@ class GlutenDynamicPartitionPruningV1SuiteAEOnDisableScan
// Same as above except AQE is off.
class GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan
- extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+ extends GlutenDynamicPartitionPruningV1SuiteAEOff {
override def sparkConf: SparkConf = {
super.sparkConf.set(GlutenConfig.COLUMNAR_FILESCAN_ENABLED.key, "false")
}
@@ -715,7 +715,7 @@ class GlutenDynamicPartitionPruningV2SuiteAEOffDisableScan
// Test DPP with project disabled by user for some reason, which can also mock
the situation
// that project is not transformable.
class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
- extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+ extends GlutenDynamicPartitionPruningV1SuiteAEOff {
override def sparkConf: SparkConf = {
super.sparkConf.set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
}
@@ -723,7 +723,7 @@ class
GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
// Same as above except whole-stage codegen is off.
class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOffDisableProject
- extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+ extends GlutenDynamicPartitionPruningV1SuiteAEOff {
override def sparkConf: SparkConf = {
super.sparkConf
.set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]