RRXXZZYY opened a new pull request, #5603: URL: https://github.com/apache/datafusion-comet/pull/5603
## Which issue does this PR close? Closes #5586. ## Rationale for this change Spark permits duplicate field names in `named_struct`, and its generated code writes those fields positionally. Routing this expression through Comet's JVM codegen dispatcher therefore preserves Spark semantics, but Arrow Java's default struct-vector factory indexes children by name. With duplicate names, it collapses the children and can reuse the wrong concrete vector type. The same allocation behavior also affects the Arrow FFI import path. ## What changes are included in this PR? - Route duplicate-name `CreateNamedStruct` expressions through the JVM codegen dispatcher instead of falling the projection back to Spark. - Allocate duplicate struct children with private, unique runtime names while preserving the original duplicate names in the exported Arrow schema. - Reuse the same duplicate-safe vector factory when importing native Arrow arrays. - Add focused codegen, Arrow FFI round-trip, and Spark integration regressions, and update the expression support documentation. ## How are these changes tested? Focused Spark 4.0 validation: - `make core` - `./mvnw test -Dtest=none -Dsuites="org.apache.comet.CometCodegenSuite codegen kernel preserves duplicate struct field names positionally" -Pspark-4.0` - `./mvnw test -Dtest=none -Dsuites="org.apache.comet.vector.NativeUtilSuite importVector preserves duplicate struct fields positionally" -Pspark-4.0` - `./mvnw test -Dtest=none -Dsuites="org.apache.comet.CometExpressionSuite named_struct with duplicate field names" -Pspark-4.0` - `./mvnw spotless:check -Pspark-4.0` The focused Spark integration test and formatting check were rerun after rebasing onto the latest `upstream/main`; both completed with `BUILD SUCCESS` (1 integration test passed, 0 failed). The rebase was conflict-free and the implementation patch was unchanged apart from a documentation-comment cleanup. Validation was local on WSL/Linux with the Spark 4.0 profile. I did not run the full multi-Spark-version suite or native macOS/Windows CI matrices; those remain for project CI. AI-assisted development disclosure: I used AI tooling to assist investigation and drafting. I manually reproduced the failure, reviewed and refined the implementation, inspected the final diff, and ran the checks listed above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
