shrirangmhalgi opened a new pull request, #58321:
URL: https://github.com/apache/spark/pull/58321
### What changes were proposed in this pull request?
This PR enables DSv2 Join pushdown for the DB2 JDBC connector by:
1. Fixing `DB2SQLQueryBuilder.build()` to use the `tableOrQuery` method
instead of directly accessing `options.tableOrQuery`. The `tableOrQuery` method
returns the join subquery (when set by `withJoin()`), which is required for
join pushdown to produce correct SQL.
2. Adding `override def supportsJoin: Boolean = true` to DB2Dialect.
### Why are the changes needed?
Join pushdown was introduced in SPARK-52187 and enabled for Oracle
(SPARK-52823), PostgreSQL (SPARK-52906), and MySQL/MSSQL (SPARK-52929). DB2 was
not included in that batch despite supporting ANSI standard join syntax
natively.
Additionally, `DB2SQLQueryBuilder.build()` had a latent bug - it used
`${options.tableOrQuery}` (the raw table name) instead of `$tableOrQuery` (the
method that returns the join subquery when one is pushed). Without this fix,
enabling supportsJoin would generate incorrect SQL that ignores the pushed join
entirely. All other dialects with custom build() overrides (Oracle, MySQL,
MSSQL) correctly use the `$tableOrQuery` method.
### Does this PR introduce _any_ user-facing change?
No. Join pushdown is an internal optimization that produces the same query
results. Users reading from DB2 via JDBC with joins will see improved
performance as the join is now executed on the DB2 server instead of in Spark.
### How was this patch tested?
- Added a unit test in JDBCSuite that directly verifies DB2SQLQueryBuilder
generates correct join SQL when withJoin() is called.
- Added DB2JoinPushdownIntegrationSuite — a Docker integration test
following the exact pattern of the existing Oracle/PostgreSQL/MySQL/MSSQL join
pushdown integration suites.
- All 317 existing JDBC tests pass with zero regressions.
### Was this patch authored or co-authored using generative AI tooling?
Yes. Co-Authored using Claude Opus 4.8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
