[
https://issues.apache.org/jira/browse/SPARK-14960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15633478#comment-15633478
]
Josh Rosen commented on SPARK-14960:
------------------------------------
It turns out that {{treeAggregation}}'s extra costs in local mode were
exaggerated by a bug impacting the serialization of ClassTags, causing huge
object graph traversals and slowing things down. After fixing that other bug,
the performance delta between running and not running {{treeAggregation}} was a
lot smaller and this JIRA's optimization became questionable. I'm going to
close this as "won't fix" for now.
> Don't perform treeAggregation in local mode
> -------------------------------------------
>
> Key: SPARK-14960
> URL: https://issues.apache.org/jira/browse/SPARK-14960
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Josh Rosen
>
> I don't think that {{treeAggregation}} will help performance in {{local}}
> mode and based on measurement of some unit tests it looks like it actually
> severely harms performance in certain cases. Therefore, treeAggregate should
> fall back to plain aggregate when running in local mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
