sam-1112 opened a new issue, #7632: URL: https://github.com/apache/datafusion-comet/issues/7632
### What is the problem the feature request solves? On Spark 4.x, Comet's Rust MapSort implementation does not support maps with struct or array keys, such as MAP<STRUCT<a: INT, b: INT>, STRING> and MAP<ARRAY<INT>, STRING>. #5590 and #5802 track a JVM codegen fallback that can keep the enclosing operator in Comet while Spark's generated code evaluates MapSort. This issue tracks a separate improvement: evaluating MapSort itself in Rust for these key types, without invoking the JVM dispatcher. ### Describe the potential solution Extend CometMapSort and the existing Rust map_sort kernel to support struct and array keys whose leaf types have Spark-compatible ordering. Preserve the existing scalar fast path and use the existing fallback behavior for unsupported key types. Leave floating-point leaves and non-default string collations out of the initial scope. Add Spark parity tests that compare the ordered entries, not just map equality, and verify that MapSort runs in Rust with JVM codegen dispatch disabled. Cover nullable struct fields, nullable array elements, arrays with a common prefix and different lengths, and null or empty maps. For native shuffle, also verify that maps with the same entries in different orders get the same partition IDs as in Spark. ### Additional context Related to #5590 and #5802. The native implementation does not depend on #5802 landing first. ### Willingness to contribute None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
