andygrove commented on PR #5844: URL: https://github.com/apache/datafusion-comet/pull/5844#issuecomment-5719749129
Triage note: #5854 makes `map_from_arrays` honour `spark.sql.mapKeyDedupPolicy` natively by passing DataFusion 55's `datafusion.spark.map_key_dedup_policy` through, and deletes `MapKeyDedupPolicySupport` from `maps.scala` on the way. This PR keeps that object and routes LAST_WIN through the codegen dispatcher instead. You are both editing the same `CometMapFromArrays` declaration and rewriting the same `map_from_arrays_dedup_policy.sql` fixture, so only one can land as written. Native looks like the better outcome here, assuming DataFusion's kernel matches `ArrayBasedMapBuilder` on ordering as well as dedup. Could you and @peterxcli agree which way LAST_WIN goes before either gets more review? If #5854 wins, the dispatcher benchmark and the extra null / empty-array cases here are still worth keeping. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
