Chenzhao Guo created SPARK-22537:
------------------------------------
Summary: Aggregation of map output statistics on driver faces
single point bottleneck
Key: SPARK-22537
URL: https://issues.apache.org/jira/browse/SPARK-22537
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.2.0
Reporter: Chenzhao Guo
In adaptive execution, the map output statistics of all mappers will be
aggregated after a stage is successfully executed. Driver takes the aggregation
job while it will get slow when the number of mapper * shuffle partitions is
large.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
