[ https://issues.apache.org/jira/browse/SPARK-20715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-20715: ------------------------------------ Assignee: Josh Rosen (was: Apache Spark) > MapStatuses shouldn't be redundantly stored in both ShuffleMapStage and > MapOutputTracker > ---------------------------------------------------------------------------------------- > > Key: SPARK-20715 > URL: https://issues.apache.org/jira/browse/SPARK-20715 > Project: Spark > Issue Type: Improvement > Components: Scheduler, Shuffle > Affects Versions: 2.3.0 > Reporter: Josh Rosen > Assignee: Josh Rosen > > Today the MapOutputTracker and ShuffleMapStage both maintain their own copies > of MapStatuses. This creates the potential for bugs in case these two pieces > of state become out of sync. > I believe that we can improve our ability to reason about the code by storing > this information only in the MapOutputTracker. This can also help to reduce > driver memory consumption. > I will provide more details in my PR, where I'll walk through the detailed > arguments as to why we can take these two different metadata tracking formats > and consolidate without loss of performance or correctness. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org