urosstan-db opened a new pull request, #58958:
URL: https://github.com/apache/spark/pull/58958
### What changes were proposed in this pull request?
Fix two JDBC Data Source V2 join pushdown bugs:
- Keep reads with `partitionColumn` configured as Spark-side joins,
preserving partitioned JDBC reads. Log the reason at debug level.
- Preserve pushed-join column aliases when rewriting a partially pushed
`AVG` into `SUM` and `COUNT`.
Add SQL regressions to `JDBCV2JoinPushdownIntegrationSuiteBase`.
### Why are the changes needed?
Join pushdown creates a JDBC `query`, which cannot be combined with
`partitionColumn`. Partitioned inputs currently fail with `Options 'query' and
'partitionColumn' can not be specified together.`
Partial `AVG` normalization uses the original relation output instead of the
pushed-join aliases. This can silently aggregate the wrong column: the H2
self-join regression returns `5.413043` instead of `4.413043` without the fix.
### Does this PR introduce _any_ user-facing change?
Yes. With JDBC V2 join pushdown enabled, partitioned input queries execute
successfully using Spark-side joins, and partially pushed averages over joins
return the correct result.
### How was this patch tested?
The shared SQL tests cover partitioned reads with one and two partitions,
and compare AVG results with one partition, two partitions, and join pushdown
disabled. The AVG test is skipped when aggregate pushdown is unsupported.
Confirmed both original regressions fail before their fixes. Also reran the
revised three-way AVG test without its fix and confirmed the incorrect result
above.
Validation on the committed changes with JDK 21 and H2: all 24 tests in
`JDBCV2JoinPushdownSuite` and all 3 selected partial-pushdown tests in
`JDBCV2Suite` passed.
```sh
build/sbt 'sql/testOnly *JDBCV2JoinPushdownSuite' \
'sql/testOnly *JDBCV2Suite -- -z "partial push-down"'
```
Other databases' integration suites were not run.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex CLI 0.154.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
