peterxcli opened a new pull request, #3369: URL: https://github.com/apache/datafusion-comet/pull/3369
## Which issue does this PR close? Closes: https://github.com/apache/datafusion-comet/issues/3164 ## Rationale for this change Comet does not currently support the Spark `map_contains_key` function, causing queries using this function to fall back to Spark's JVM execution instead of running natively on DataFusion. The MapContainsKey expression checks whether a given key exists in a map. It is implemented as a runtime-replaceable expression that internally uses ArrayContains on the map's keys to perform the lookup. Supporting this expression would allow more Spark workloads to benefit from Comet's native acceleration. ## What changes are included in this PR? - query plan serde to map `MapContainsKey` -> `CometMapContainsKey` - implement `CometMapContainsKey` to convert the expr to `array_has(map_keys(map), key)` - ref: https://github.com/apache/spark/blob/04b821c69e85be5f51a1270b3a9a4155afdb5334/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L239 - update spark expr supporting list - btw, I saw https://github.com/apache/datafusion-comet/pull/1788 had implemented the `map_keys` function, so I also mark it as done. ## How are these changes tested? - gen map type data and use that to verify the correctness of `map_contains_key` - Empty-map tests empty map tests in spark: - https://github.com/apache/spark/blob/04b821c69e85be5f51a1270b3a9a4155afdb5334/sql/core/src/test/scala/org/apache/spark/sql/ParametersSuite.scala#L609-L618 - https://github.com/apache/spark/blob/04b821c69e85be5f51a1270b3a9a4155afdb5334/sql/core/src/test/scala/org/apache/spark/sql/ParametersSuite.scala#L1663-L1672 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
