[ https://issues.apache.org/jira/browse/SPARK-31973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372286#comment-17372286 ]
Apache Spark commented on SPARK-31973: -------------------------------------- User 'shipra-a' has created a pull request for this issue: https://github.com/apache/spark/pull/33161 > Add ability to disable Sort,Spill in Partial aggregation > --------------------------------------------------------- > > Key: SPARK-31973 > URL: https://issues.apache.org/jira/browse/SPARK-31973 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.0.0 > Reporter: Karuppayya > Priority: Major > > In case of HashAggregation, a partial aggregation(update) is done followed by > final aggregation(merge) > During partial aggregation we sort and spill to disk everytime, when the fast > Map(when enabled) and UnsafeFixedWidthAggregationMap gets exhausted > *When the cardinality of grouping column is close to the total number of > records being processed*, the sorting of data spilling to disk is not > required, since it is kind of no-op and we can directly use in Final > aggregation. > When the user is aware of nature of data, currently he has no control over > disabling this sort, spill operation. > This is similar to following issue in Hive: > https://issues.apache.org/jira/browse/HIVE-223 > https://issues.apache.org/jira/browse/HIVE-291 > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org