rdtr opened a new pull request, #13588: URL: https://github.com/apache/gluten/pull/13588
Depends on #13520. Only the last commit belongs to this PR: https://github.com/rdtr/incubator-gluten/commit/d7ea1639ff49631de4a43747c3c783bcc5c0db95 ## What changes were proposed in this pull request? Spark 4.3 (SPARK-58265) can derive DPP partition values from the full rows of an existing broadcast, through `ProjectedBroadcastValueSubqueryExec`. It is off by default, behind the internal conf `spark.sql.optimizer.dynamicPartitionPruning.broadcastProjection.enabled`. #13520 excludes its 24 tests (4 test names over the V1 and V2 suites) with a generic comment. Projection pruning isn't supported with columnar broadcasts, and this PR doesn't add it. With the conf on and AQE on, the broadcast is a columnar `BuildSideRelation`. Spark's projection reads a `HashedRelation`, hits an error and marks the domain unavailable, so every partition is scanned and answers stay correct. With AQE off, `RewriteSubqueryBroadcast` doesn't touch the new node. Its exchange stays a row `BroadcastExchangeExec`, so pruning works but the build side is broadcast a second time instead of reusing the join's columnar broadcast. The 24 tests stay excluded, now with the actual reason. 18 are on AQE-on suites and depend on pruning values that Gluten doesn't produce. 6 are the composite-key test on AQE-off suites, which stops at the `lastAttemptValueForDataset` assert. I didn't trace which Gluten node makes that return None, and the asserts after it didn't run. This adds two Gluten tests to the V1 suites, one per behavior, with the conf on and ANSI off so the plan stays native: - `GlutenDynamicPartitionPruningV1SuiteAEOn`: "SPARK-58265: projected broadcast values fail open on a columnar broadcast" checks the answer, a native scan, an unavailable domain, `projectionDisabled` = 1 and `projectionErrors` = 1. - `GlutenDynamicPartitionPruningV1SuiteAEOff`: "SPARK-58265: projected broadcast values come from a separate row broadcast" checks the answer, the projected values, a row exchange under the subquery and no `ReusedExchangeExec`. The first test is excluded on `AEOnDisableScan`, since it asserts the native scan. Supporting this would take a Spark 4.3 shim, because the node only exists in 4.3, and it has to keep Spark's node, because `InSubqueryExec` only recognizes the unavailable result on that type. The child would expose the columnar broadcast as a lazily built `HashedRelation` and implement `BroadcastExchangeLike` for Spark's size check, and the exchange would need rewriting to columnar for reuse without AQE. That's roughly 100+ lines for an off-by-default optimization, so I left it out. Let me know if you'd want it. ## How was this patch tested? Spark 4.3.0, all 12 `GlutenDynamicPartitionPruning*` suites with `SPARK_ANSI_SQL_MODE=false`: 462 passed, 0 failed, 67 ignored. Before this change, with the 24 excludes removed, the same suites had 24 failures. Only `gluten-ut/spark43` test files change. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
