shivadarshan-devadiga opened a new pull request, #13620:
URL: https://github.com/apache/gluten/pull/13620
## What changes are proposed in this pull request?
Fixes #12857
Three DPP suites carry `V1` in their name but extend
`GlutenDynamicPartitionPruningV2SuiteAEOff`, so they re-run the V2 scenario
instead of covering the V1 DataSource path:
- `GlutenDynamicPartitionPruningV1SuiteAEOffDisableScan`
- `GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOnDisableProject`
- `GlutenDynamicPartitionPruningV1SuiteAEOffWSCGOffDisableProject`
`V1SuiteAEOffDisableScan` sets `COLUMNAR_FILESCAN_ENABLED=false`, which has
no effect on the V2 batch scan. The two `V1...DisableProject` suites were
identical to their `V2...` counterparts. The AQE-on variant,
`V1SuiteAEOnDisableScan`, already extends the V1 base. This PR changes the
three suites to extend `GlutenDynamicPartitionPruningV1SuiteAEOff` in all five
modules (`gluten-ut/spark34`, `spark35`, `spark40`, `spark41`, `spark42`).
Running them on the V1 base exposed one test that never ran with the file
scan disabled: the Gluten copy of `Subquery reuse across the whole plan` in
`spark34`/`spark35`. It expects 3 subqueries and 2 reuses, because Gluten's
native scan also pushes down the subquery filter
(`FilterHandler#applyFilterPushdownToScan`). With the scan falling back to
vanilla Spark, the plan has vanilla Spark's 2 and 1, so the test failed. The
test now takes its expectation from `COLUMNAR_FILESCAN_ENABLED`: 3/2 when the
file scan is offloaded, 2/1 when it is disabled. It also asserts that the plan
has a `FileSourceScanExecTransformer` exactly when the config says so, so a
scan that silently falls back in the default suite still fails the test. Spark
4.x doesn't have this copy; there Spark's own test runs, and it expects 2/1.
ClickHouse excludes this Gluten test on `V1SuiteAEOff` in spark35.
`V1SuiteAEOffDisableScan` now inherits it, so the same exclusion is added to
that entry. The other backends and versions inherit nothing new that they
exclude on the parent.
## How was this patch tested?
- Ran `GlutenDynamicPartitionPruningV1SuiteAEOff`,
`V1SuiteAEOffDisableScan`, `V1SuiteAEOffWSCGOnDisableProject` and
`V1SuiteAEOffWSCGOffDisableProject` locally on the Velox backend, on each Spark
version, with ANSI off as in the regular CI:
| Spark | Tests passed | Failed |
|---|---|---|
| 3.4 | 152 | 0 |
| 3.5 | 152 | 0 |
| 4.0 | 152 | 0 |
| 4.1 | 152 | 0 |
| 4.2 | 152 | 0 |
- On 3.4/3.5 the base suite takes the 3/2 branch of `Subquery reuse across
the whole plan` and `V1SuiteAEOffDisableScan` takes the 2/1 branch. Before the
test change, `V1SuiteAEOffDisableScan` failed it with `ArrayBuffer(...) had
size 2 instead of expected size 3`.
- `spotless:check` passes for `gluten-ut/spark34` and `gluten-ut/spark35`.
- The ClickHouse and Bolt backends were not run locally; CI covers them. The
three `Disable*` suites are the first runs of these V1 paths, so if a backend's
CI flags one of them, the inherited V1 tests are the place to look.
## Was this patch authored or co-authored using generative AI tooling?
Co-authored by: Claude Sonnet 4.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]