XdithyX opened a new pull request, #56486:
URL: https://github.com/apache/spark/pull/56486
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR:
https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a
faster review.
7. If you want to add a new configuration, please read the guideline first
for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the
guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
This PR extends the DataSource V2 join pushdown API so that Spark can pass
already-pushed table sample information to connectors when attempting join
pushdown.
Before this change, Spark blocked join pushdown when either side of the join
had a real pushed sample, because the merged join scan builder had no way to
know about that sample. Allowing join pushdown in that state would have
silently dropped the sample and changed query results.
This PR adds:
1. A new sample-aware `SupportsPushDownJoin.pushDownJoin(...)` overload that
receives:
- `leftSample`
- `rightSample`
2. A public `SupportsPushDownJoin.TableSample` record containing:
- `lowerBound`
- `upperBound`
- `withReplacement`
- `seed`
- `sampleMethod`
3. Default compatibility behavior for existing connectors:
- If there is no sample, or only a no-op sample, Spark delegates to the
existing join pushdown method.
- If there is a real pushed sample, the default implementation returns
`false`, so existing connectors do not accidentally drop the sample.
4. Optimizer wiring in `V2ScanRelationPushDown` to pass pushed sample
information from the left and right scan builders into the new sample-aware
join pushdown API.
5. JDBC join pushdown support for pushed samples:
- `JDBCScanBuilder` now preserves table sample clauses in the left and
right pushed join subqueries.
- If a pushed sample cannot be represented as a JDBC table sample clause,
join pushdown is rejected.
6. Test coverage for:
- left-side pushed sample with join pushdown
- right-side pushed sample with join pushdown
- both-side pushed samples with join pushdown
- old join pushdown API compatibility
- rejection of real pushed samples by connectors that do not override the
new API
- JDBC SQL generation preserving table sample clauses in pushed join sides
### Why are the changes needed?
SPARK-55978 added table sample pushdown support and intentionally blocked
composing pushed samples with join pushdown. That block was necessary because
`SupportsPushDownJoin` did not receive sample information. If Spark pushed the
join after pushing a sample, the connector could build a joined scan without
knowing that one side had already been sampled.
This PR fixes that limitation.
With this change, connectors that support both sample pushdown and join
pushdown can make an explicit decision:
- return `true` when they can correctly execute the sampled join
- return `false` when they cannot
Existing connectors remain safe because the new default method rejects real
pushed samples unless the connector explicitly opts in by overriding the new
overload.
### Does this PR introduce _any_ user-facing change?
Yes, for DataSource V2 connectors that implement the new sample-aware join
pushdown API.
Previously, Spark did not push down joins when either side had a real pushed
sample. After this change, Spark can push down such joins if the connector
explicitly supports the sample and join combination.
For existing connectors that only implement the old join pushdown API,
behavior remains safe: Spark will not push down a join with a real pushed
sample through the default implementation.
### How was this patch tested?
Added and updated tests in:
- `DataSourceV2TableSampleSuite`
- `JDBCSuite`
- in-memory test connector/catalog fixtures for sample-aware and legacy join
pushdown behavior
Ran:
```bash
build/sbt 'sql/testOnly *DataSourceV2TableSampleSuite' 'sql/testOnly
org.apache.spark.sql.jdbc.JDBCSuite -- -z SPARK-56504'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex GPT 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
