This is an automated email from the ASF dual-hosted git repository.

LuciferYang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git


The following commit(s) were added to refs/heads/main by this push:
     new caa83df7fb [GLUTEN-12857][TEST] Fix V1 DPP suites extending the V2 
base (#13620)
caa83df7fb is described below

commit caa83df7fb6569827e1a70c7a85a022e7b08d983
Author: Shivadarshan Devadiga <[email protected]>
AuthorDate: Thu Nov 12 18:11:38 2026 +0530

    [GLUTEN-12857][TEST] Fix V1 DPP suites extending the V2 base (#13620)
    
    ## What changes are proposed in this pull request?
    
    Fixes #12857
    
    Three DPP suites carry `V1` in their name but extend 
`GlutenDynamicPartitionPruningV2SuiteAEOff`, so they re-run the V2 scenario 
instead of covering the V1 DataSource path:
    
    - `GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan`
    - `GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject`
    - `GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOffDisableProject`
    
    `V1SuiteAEOffDisableScan` sets `COLUMNAR_FILESCAN_ENABLED=false`, which has 
no effect on the V2 batch scan. The two `V1...DisableProject` suites were 
identical to their `V2...` counterparts. The AQE-on variant, 
`V1SuiteAEOnDisableScan`, already extends the V1 base. This PR changes the 
three suites to extend `GlutenDynamicPartitionPruningV1SuiteAEOff` in all five 
modules (`gluten-ut/spark34`, `spark35`, `spark40`, `spark41`, `spark42`).
    
    Running them on the V1 base exposed one test that never ran with the file 
scan disabled: the Gluten copy of `Subquery reuse across the whole plan` in 
`spark34`/`spark35`. It expects 3 subqueries and 2 reuses, because Gluten's 
native scan also pushes down the subquery filter 
(`FilterHandler#applyFilterPushdownToScan`). With the scan falling back to 
vanilla Spark, the plan has vanilla Spark's 2 and 1, so the test failed. The 
test now takes its expectation from `COLUMNAR_FILESCAN_ENABLED [...]
    
    ClickHouse excludes this Gluten test on `V1SuiteAEOff` in spark35. 
`V1SuiteAEOffDisableScan` now inherits it, so the same exclusion is added to 
that entry. The other backends and versions inherit nothing new that they 
exclude on the parent.
    
    ## How was this patch tested?
    
    - Ran `GlutenDynamicPartitionPruningV1SuiteAEOff`, 
`V1SuiteAEOffDisableScan`, `V1SuiteAEOffWSCGOnDisableProject` and 
`V1SuiteAEOffWSCGOffDisableProject` locally on the Velox backend, on each Spark 
version, with ANSI off as in the regular CI:
    
      | Spark | Tests passed | Failed |
      |---|---|---|
      | 3.4 | 152 | 0 |
      | 3.5 | 152 | 0 |
      | 4.0 | 152 | 0 |
      | 4.1 | 152 | 0 |
      | 4.2 | 152 | 0 |
    
    - On 3.4/3.5 the base suite takes the 3/2 branch of `Subquery reuse across 
the whole plan` and `V1SuiteAEOffDisableScan` takes the 2/1 branch. Before the 
test change, `V1SuiteAEOffDisableScan` failed it with `ArrayBuffer(...) had 
size 2 instead of expected size 3`.
    - `spotless:check` passes for `gluten-ut/spark34` and `gluten-ut/spark35`.
    - The ClickHouse and Bolt backends were not run locally; CI covers them. 
The three `Disable*` suites are the first runs of these V1 paths, so if a 
backend's CI flags one of them, the inherited V1 tests are the place to look.
    
    ## Was this patch authored or co-authored using generative AI tooling?
    Co-authored by: Claude Sonnet 4.5
---
 .../sql/GlutenDynamicPartitionPruningSuite.scala   | 23 ++++++++++++++++------
 .../utils/clickhouse/ClickHouseTestSettings.scala  |  1 +
 .../sql/GlutenDynamicPartitionPruningSuite.scala   | 23 ++++++++++++++++------
 .../sql/GlutenDynamicPartitionPruningSuite.scala   |  6 +++---
 .../sql/GlutenDynamicPartitionPruningSuite.scala   |  6 +++---
 .../sql/GlutenDynamicPartitionPruningSuite.scala   |  6 +++---
 6 files changed, 44 insertions(+), 21 deletions(-)

diff --git 
a/gluten-ut/spark34/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
 
b/gluten-ut/spark34/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
index 3b2be3c72e..5d7c18d082 100644
--- 
a/gluten-ut/spark34/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
+++ 
b/gluten-ut/spark34/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
@@ -722,12 +722,23 @@ class GlutenDynamicPartitionPruningV1SuiteAEOff
           case rs: ReusedSubqueryExec => rs.child.id
         }
 
-        // By default Gluten pushes more filters than vanilla Spark.
+        // By default Gluten pushes more filters than vanilla Spark. When the 
file scan is offloaded
+        // (COLUMNAR_FILESCAN_ENABLED), the native scan also takes the 
subquery filter, which adds
+        // one subquery and one reuse. With the file scan disabled, the plan 
has vanilla Spark's
+        // counts.
         //
         // See also 
org.apache.gluten.execution.FilterHandler#applyFilterPushdownToScan
         // See also DynamicPartitionPruningSuite.scala:1362
-        assert(subqueryIds.size == 3, "Whole plan subquery reusing not working 
correctly")
-        assert(reusedSubqueryIds.size == 2, "Whole plan subquery reusing not 
working correctly")
+        val scanOffloaded = GlutenConfig.get.enableColumnarFileScan
+        val nativeScan = plan.collectWithSubqueries { case s: 
FileSourceScanExecTransformer => s }
+        assert(nativeScan.nonEmpty == scanOffloaded, "Unexpected file scan 
offloading")
+        val (expectedSubqueries, expectedReused) = if (scanOffloaded) (3, 2) 
else (2, 1)
+        assert(
+          subqueryIds.size == expectedSubqueries,
+          "Whole plan subquery reusing not working correctly")
+        assert(
+          reusedSubqueryIds.size == expectedReused,
+          "Whole plan subquery reusing not working correctly")
         assert(
           reusedSubqueryIds.forall(subqueryIds.contains(_)),
           "ReusedSubqueryExec should reuse an existing subquery")
@@ -806,7 +817,7 @@ class GlutenDynamicPartitionPruningV1SuiteAEOnDisableScan
 
 // Same as above except AQE is off.
 class GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan
-  extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+  extends GlutenDynamicPartitionPruningV1SuiteAEOff {
   override def sparkConf: SparkConf = {
     super.sparkConf.set(GlutenConfig.COLUMNAR_FILESCAN_ENABLED.key, "false")
   }
@@ -832,7 +843,7 @@ class GlutenDynamicPartitionPruningV2SuiteAEOffDisableScan
 // Test DPP with project disabled by user for some reason, which can also mock 
the situation
 // that project is not transformable.
 class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
-  extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+  extends GlutenDynamicPartitionPruningV1SuiteAEOff {
   override def sparkConf: SparkConf = {
     super.sparkConf.set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
   }
@@ -840,7 +851,7 @@ class 
GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
 
 // Same as above except whole-stage codegen is off.
 class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOffDisableProject
-  extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+  extends GlutenDynamicPartitionPruningV1SuiteAEOff {
   override def sparkConf: SparkConf = {
     super.sparkConf
       .set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
diff --git 
a/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/clickhouse/ClickHouseTestSettings.scala
 
b/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/clickhouse/ClickHouseTestSettings.scala
index 788a5c82ef..ca81410bb5 100644
--- 
a/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/clickhouse/ClickHouseTestSettings.scala
+++ 
b/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/clickhouse/ClickHouseTestSettings.scala
@@ -769,6 +769,7 @@ class ClickHouseTestSettings extends BackendTestSettings {
   enableSuite[GlutenDynamicPartitionPruningV1SuiteAEOff]
     .excludeGlutenTest("Subquery reuse across the whole plan")
   enableSuite[GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan]
+    .excludeGlutenTest("Subquery reuse across the whole plan")
   enableSuite[GlutenDynamicPartitionPruningV1SuiteAEOn]
   enableSuite[GlutenDynamicPartitionPruningV1SuiteAEOnDisableScan]
   enableSuite[GlutenDynamicPartitionPruningV2SuiteAEOff]
diff --git 
a/gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
 
b/gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
index ab34d46f67..d6293f7ce6 100644
--- 
a/gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
+++ 
b/gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
@@ -619,12 +619,23 @@ class GlutenDynamicPartitionPruningV1SuiteAEOff
           case rs: ReusedSubqueryExec => rs.child.id
         }
 
-        // By default Gluten pushes more filters than vanilla Spark.
+        // By default Gluten pushes more filters than vanilla Spark. When the 
file scan is offloaded
+        // (COLUMNAR_FILESCAN_ENABLED), the native scan also takes the 
subquery filter, which adds
+        // one subquery and one reuse. With the file scan disabled, the plan 
has vanilla Spark's
+        // counts.
         //
         // See also 
org.apache.gluten.execution.FilterHandler#applyFilterPushdownToScan
         // See also DynamicPartitionPruningSuite.scala:1362
-        assert(subqueryIds.size == 3, "Whole plan subquery reusing not working 
correctly")
-        assert(reusedSubqueryIds.size == 2, "Whole plan subquery reusing not 
working correctly")
+        val scanOffloaded = GlutenConfig.get.enableColumnarFileScan
+        val nativeScan = plan.collectWithSubqueries { case s: 
FileSourceScanExecTransformer => s }
+        assert(nativeScan.nonEmpty == scanOffloaded, "Unexpected file scan 
offloading")
+        val (expectedSubqueries, expectedReused) = if (scanOffloaded) (3, 2) 
else (2, 1)
+        assert(
+          subqueryIds.size == expectedSubqueries,
+          "Whole plan subquery reusing not working correctly")
+        assert(
+          reusedSubqueryIds.size == expectedReused,
+          "Whole plan subquery reusing not working correctly")
         assert(
           reusedSubqueryIds.forall(subqueryIds.contains(_)),
           "ReusedSubqueryExec should reuse an existing subquery")
@@ -746,7 +757,7 @@ class GlutenDynamicPartitionPruningV1SuiteAEOnDisableScan
 
 // Same as above except AQE is off.
 class GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan
-  extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+  extends GlutenDynamicPartitionPruningV1SuiteAEOff {
   override def sparkConf: SparkConf = {
     super.sparkConf.set(GlutenConfig.COLUMNAR_FILESCAN_ENABLED.key, "false")
   }
@@ -772,7 +783,7 @@ class GlutenDynamicPartitionPruningV2SuiteAEOffDisableScan
 // Test DPP with project disabled by user for some reason, which can also mock 
the situation
 // that project is not transformable.
 class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
-  extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+  extends GlutenDynamicPartitionPruningV1SuiteAEOff {
   override def sparkConf: SparkConf = {
     super.sparkConf.set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
   }
@@ -780,7 +791,7 @@ class 
GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
 
 // Same as above except whole-stage codegen is off.
 class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOffDisableProject
-  extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+  extends GlutenDynamicPartitionPruningV1SuiteAEOff {
   override def sparkConf: SparkConf = {
     super.sparkConf
       .set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
diff --git 
a/gluten-ut/spark40/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
 
b/gluten-ut/spark40/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
index 9f4e18fb82..aee71ec88e 100644
--- 
a/gluten-ut/spark40/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
+++ 
b/gluten-ut/spark40/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
@@ -687,7 +687,7 @@ class GlutenDynamicPartitionPruningV1SuiteAEOnDisableScan
 
 // Same as above except AQE is off.
 class GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan
-  extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+  extends GlutenDynamicPartitionPruningV1SuiteAEOff {
   override def sparkConf: SparkConf = {
     super.sparkConf.set(GlutenConfig.COLUMNAR_FILESCAN_ENABLED.key, "false")
   }
@@ -713,7 +713,7 @@ class GlutenDynamicPartitionPruningV2SuiteAEOffDisableScan
 // Test DPP with project disabled by user for some reason, which can also mock 
the situation
 // that project is not transformable.
 class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
-  extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+  extends GlutenDynamicPartitionPruningV1SuiteAEOff {
   override def sparkConf: SparkConf = {
     super.sparkConf.set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
   }
@@ -721,7 +721,7 @@ class 
GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
 
 // Same as above except whole-stage codegen is off.
 class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOffDisableProject
-  extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+  extends GlutenDynamicPartitionPruningV1SuiteAEOff {
   override def sparkConf: SparkConf = {
     super.sparkConf
       .set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
diff --git 
a/gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
 
b/gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
index 2a1a2c55fe..0ac03f1b18 100644
--- 
a/gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
+++ 
b/gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
@@ -689,7 +689,7 @@ class GlutenDynamicPartitionPruningV1SuiteAEOnDisableScan
 
 // Same as above except AQE is off.
 class GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan
-  extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+  extends GlutenDynamicPartitionPruningV1SuiteAEOff {
   override def sparkConf: SparkConf = {
     super.sparkConf.set(GlutenConfig.COLUMNAR_FILESCAN_ENABLED.key, "false")
   }
@@ -715,7 +715,7 @@ class GlutenDynamicPartitionPruningV2SuiteAEOffDisableScan
 // Test DPP with project disabled by user for some reason, which can also mock 
the situation
 // that project is not transformable.
 class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
-  extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+  extends GlutenDynamicPartitionPruningV1SuiteAEOff {
   override def sparkConf: SparkConf = {
     super.sparkConf.set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
   }
@@ -723,7 +723,7 @@ class 
GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
 
 // Same as above except whole-stage codegen is off.
 class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOffDisableProject
-  extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+  extends GlutenDynamicPartitionPruningV1SuiteAEOff {
   override def sparkConf: SparkConf = {
     super.sparkConf
       .set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
diff --git 
a/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
 
b/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
index 84a9107852..418dde7138 100644
--- 
a/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
+++ 
b/gluten-ut/spark42/src/test/scala/org/apache/spark/sql/GlutenDynamicPartitionPruningSuite.scala
@@ -689,7 +689,7 @@ class GlutenDynamicPartitionPruningV1SuiteAEOnDisableScan
 
 // Same as above except AQE is off.
 class GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan
-  extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+  extends GlutenDynamicPartitionPruningV1SuiteAEOff {
   override def sparkConf: SparkConf = {
     super.sparkConf.set(GlutenConfig.COLUMNAR_FILESCAN_ENABLED.key, "false")
   }
@@ -715,7 +715,7 @@ class GlutenDynamicPartitionPruningV2SuiteAEOffDisableScan
 // Test DPP with project disabled by user for some reason, which can also mock 
the situation
 // that project is not transformable.
 class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
-  extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+  extends GlutenDynamicPartitionPruningV1SuiteAEOff {
   override def sparkConf: SparkConf = {
     super.sparkConf.set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")
   }
@@ -723,7 +723,7 @@ class 
GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject
 
 // Same as above except whole-stage codegen is off.
 class GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOffDisableProject
-  extends GlutenDynamicPartitionPruningV2SuiteAEOff {
+  extends GlutenDynamicPartitionPruningV1SuiteAEOff {
   override def sparkConf: SparkConf = {
     super.sparkConf
       .set(GlutenConfig.COLUMNAR_PROJECT_ENABLED.key, "false")


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to