[
https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182122#comment-14182122
]
Patrick Wendell commented on SPARK-1239:
----------------------------------------
Hey Kostas - there are a few other bugs that required a refactoring to solve
them that will subsume this issue, so I think that's why [~joshrosen] is
grabbing it. This change will involve fairly significant surgery to a very
complex part of Spark, so it might not be the best beginner issue. If Josh's
other changes don't end up fixing this, we can leave this open as an
independent issue and see if it can be done surgically.
> Don't fetch all map output statuses at each reducer during shuffles
> -------------------------------------------------------------------
>
> Key: SPARK-1239
> URL: https://issues.apache.org/jira/browse/SPARK-1239
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, Spark Core
> Affects Versions: 1.0.2, 1.1.0
> Reporter: Patrick Wendell
> Assignee: Josh Rosen
>
> Instead we should modify the way we fetch map output statuses to take both a
> mapper and a reducer - or we should just piggyback the statuses on each task.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
