minni31 opened a new pull request, #12766: URL: https://github.com/apache/gluten/pull/12766
## What changes are proposed in this pull request? This PR offloads `EmptyRelationExec` to the Velox backend so that empty relations are executed columnarly instead of forcing a fallback to vanilla row execution. `EmptyRelationExec` is a leaf node that AQE's *Propagate Empty Relations* optimization creates on Spark 4.0+ when it proves a subtree produces no output (e.g. `WHERE 1 = 0`, a join with an empty side, or a runtime-empty AQE stage). Because its default `supportsColumnar` is `false`, Gluten currently wraps it in `ColumnarToRow` / `RowToColumnar` transitions even though it returns zero rows. This PR adds `EmptyRelationExecTransformer`, a columnar leaf that returns an empty `RDD[ColumnarBatch]`, eliminating those transitions. `EmptyRelationExec` only exists on Spark 4.0+ (SPARK-47217), so the node is never referenced from version-agnostic modules. Detection is routed through a new `SparkShims.isEmptyRelationExec`, which defaults to `false` and is overridden only in the Spark 4.0 and 4.1 shims. The shared `OffloadOthers` rule and the `SparkPlanExecApi` trait therefore continue to compile unchanged against Spark 3.3–3.5. The offload is gated by a new config `spark.gluten.sql.columnar.emptyRelation` (default `true`). The Velox backend implements `isSupportEmptyRelationExec`; other backends inherit the trait default and keep vanilla execution. ## How was this patch tested? - New `VeloxEmptyRelationSuite`: empty-result correctness across various schemas, empty `UNION ALL`, AQE propagation through joins/aggregations, and side-by-side parity with vanilla Spark (asserted on all supported Spark versions). Plan-shape assertions (transformer present, no residual `EmptyRelationExec`) and the config-disabled negative case are gated to Spark 4.0+, where the node exists. - Re-enabled the upstream `SPARK-35585` AQE test on Spark 4.0/4.1 with a Gluten-aware assertion that accepts either `EmptyRelationExec` or `EmptyRelationExecTransformer`. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: GitHub Copilot (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
