shivadarshan-devadiga opened a new pull request, #13620:
URL: https://github.com/apache/gluten/pull/13620

   ## What changes are proposed in this pull request?
   
   Fixes #12857
   
   Three DPP suites carry `V1` in their name but extend 
`GlutenDynamicPartitionPruningV2SuiteAEOff`, so they re-run the V2 scenario 
instead of covering the V1 DataSource path:
   
   - `GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan`
   - `GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject`
   - `GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOffDisableProject`
   
   `V1SuiteAEOffDisableScan` sets `COLUMNAR_FILESCAN_ENABLED=false`, which has 
no effect on the V2 batch scan. The two `V1...DisableProject` suites were 
identical to their `V2...` counterparts. The AQE-on variant, 
`V1SuiteAEOnDisableScan`, already extends the V1 base. This PR changes the 
three suites to extend `GlutenDynamicPartitionPruningV1SuiteAEOff` in all five 
modules (`gluten-ut/spark34`, `spark35`, `spark40`, `spark41`, `spark42`).
   
   Running them on the V1 base exposed one test that never ran with the file 
scan disabled: the Gluten copy of `Subquery reuse across the whole plan` in 
`spark34`/`spark35`. It expects 3 subqueries and 2 reuses, because Gluten's 
native scan also pushes down the subquery filter 
(`FilterHandler#applyFilterPushdownToScan`). With the scan falling back to 
vanilla Spark, the plan has vanilla Spark's 2 and 1, so the test failed. The 
test now takes its expectation from `COLUMNAR_FILESCAN_ENABLED`: 3/2 when the 
file scan is offloaded, 2/1 when it is disabled. It also asserts that the plan 
has a `FileSourceScanExecTransformer` exactly when the config says so, so a 
scan that silently falls back in the default suite still fails the test. Spark 
4.x doesn't have this copy; there Spark's own test runs, and it expects 2/1.
   
   ClickHouse excludes this Gluten test on `V1SuiteAEOff` in spark35. 
`V1SuiteAEOffDisableScan` now inherits it, so the same exclusion is added to 
that entry. The other backends and versions inherit nothing new that they 
exclude on the parent.
   
   ## How was this patch tested?
   
   - Ran `GlutenDynamicPartitionPruningV1SuiteAEOff`, 
`V1SuiteAEOffDisableScan`, `V1SuiteAEOffWSCGOnDisableProject` and 
`V1SuiteAEOffWSCGOffDisableProject` locally on the Velox backend, on each Spark 
version, with ANSI off as in the regular CI:
   
     | Spark | Tests passed | Failed |
     |---|---|---|
     | 3.4 | 152 | 0 |
     | 3.5 | 152 | 0 |
     | 4.0 | 152 | 0 |
     | 4.1 | 152 | 0 |
     | 4.2 | 152 | 0 |
   
   - On 3.4/3.5 the base suite takes the 3/2 branch of `Subquery reuse across 
the whole plan` and `V1SuiteAEOffDisableScan` takes the 2/1 branch. Before the 
test change, `V1SuiteAEOffDisableScan` failed it with `ArrayBuffer(...) had 
size 2 instead of expected size 3`.
   - `spotless:check` passes for `gluten-ut/spark34` and `gluten-ut/spark35`.
   - The ClickHouse and Bolt backends were not run locally; CI covers them. The 
three `Disable*` suites are the first runs of these V1 paths, so if a backend's 
CI flags one of them, the inherited V1 tests are the place to look.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   Co-authored by: Claude Sonnet 4.5
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to