sam-1112 opened a new issue, #7632:
URL: https://github.com/apache/datafusion-comet/issues/7632

   ### What is the problem the feature request solves?
   
   On Spark 4.x, Comet's Rust MapSort implementation does not support maps with 
struct or array keys, such as MAP<STRUCT<a: INT, b: INT>, STRING> and 
MAP<ARRAY<INT>, STRING>.
   
   #5590 and #5802 track a JVM codegen fallback that can keep the enclosing 
operator in Comet while Spark's generated code evaluates MapSort. This issue 
tracks a separate improvement: evaluating MapSort itself in Rust for these key 
types, without invoking the JVM dispatcher.
   
   ### Describe the potential solution
   
   Extend CometMapSort and the existing Rust map_sort kernel to support struct 
and array keys whose leaf types have Spark-compatible ordering. Preserve the 
existing scalar fast path and use the existing fallback behavior for 
unsupported key types. Leave floating-point leaves and non-default string 
collations out of the initial scope.
   
   Add Spark parity tests that compare the ordered entries, not just map 
equality, and verify that MapSort runs in Rust with JVM codegen dispatch 
disabled. Cover nullable struct fields, nullable array elements, arrays with a 
common prefix and different lengths, and null or empty maps. For native 
shuffle, also verify that maps with the same entries in different orders get 
the same partition IDs as in Spark.
   
   ### Additional context
   
   Related to #5590 and #5802. The native implementation does not depend on 
#5802 landing first.
   
   ### Willingness to contribute
   
   None


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to