LinSimon-901101 opened a new pull request, #5889: URL: https://github.com/apache/datafusion-comet/pull/5889
## Which issue does this PR close? Closes #5834. ## Rationale for this change Spark's merged subqueries produce struct-typed scalar results that currently force the consuming projection to fall back to Spark. Spark 4.2's MergeSubplans broadens this to queries without explicit subqueries, potentially taking Union and downstream aggregates off Comet as well. ## What changes are included in this PR? - Support non-empty struct scalar-subquery results with distinct field names and supported scalar or nested-struct fields. - Transfer materialized results through Arrow IPC and cache owned native values within the expression's execution context. - Preserve NULL-struct semantics, field order and planned metadata, while leaving existing non-struct paths unchanged. - Add SQL, optimizer-plan and Rust regression tests, and document the supported scope. Duplicate-name structs and unsupported field types retain fallback. This PR does not claim to resolve #4949 or the separate native shuffle/Parquet issues in #5605 and #5783. Existing malformed-UTF-8 limitations still apply. ## How are these changes tested? Local validation on macOS arm64: - Before the upstream merge: focused regressions passed on Spark 3.4.3, 3.5.9, 4.0.4 and 4.1.3. - After the upstream merge, at `918202206`: native build succeeded; Spark 4.2.0 focused tests passed (3 tests), and Rust scalar-subquery tests passed (5 tests). Scalastyle and Spotless also passed. - Red/green comparison on Spark 4.1.3: the same two regressions failed against the original base because of unsupported structs/projection fallback, then passed with the implementation. The tests compare Spark results and assert native operator coverage, including the consuming projection and the Spark 4.2 Union/aggregate reproducer. The latter uses the existing test setup with Spark-to-columnar input conversion enabled. Clippy, Apache RAT and Scalafix have not yet been verified locally. Full CI validation remains pending. No benchmarks were run, and no measured performance improvement is claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
