rishvin commented on PR #2381: URL: https://github.com/apache/datafusion-comet/pull/2381#issuecomment-3282348834
> > Hi @comphead / @mbutrovich : Spark does have `MapSort` internal expression. This piece was required to support grouping on map type. Spark applies it internally. Context here: [#1941 (comment)](https://github.com/apache/datafusion-comet/issues/1941#issuecomment-3120621876) > > Thanks @rishvin the PR was saying scalar `map_sort` scalar function but this is MapSort operator to transform DataFrame . I think it would be nice to provide a test case where Spark falls back on MapSort so we can start from there? Hi @comphead : This PR was split from my first PR: https://github.com/apache/datafusion-comet/pull/2221. The mentioned PR includes integration tests in the file `CometAggregationSuite.scala`. Without the required changes, tests will fail. Those tests cases verify the physical plan along with results and without required changes they will have spark's `HashAggregate` instead of `CometHashAggregate`. We fallback when we see grouping on map type here - https://github.com/apache/datafusion-comet/blob/3b29cb968a0c89f9fc886d2aa785d7e0c3914a4c/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala#L1439. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
