[ https://issues.apache.org/jira/browse/SPARK-25341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-25341: ---------------------------------- Affects Version/s: (was: 2.4.0) 3.0.0 > Support rolling back a shuffle map stage and re-generate the shuffle files > -------------------------------------------------------------------------- > > Key: SPARK-25341 > URL: https://issues.apache.org/jira/browse/SPARK-25341 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 3.0.0 > Reporter: Wenchen Fan > Priority: Major > > This is a follow up of https://issues.apache.org/jira/browse/SPARK-23243 > To completely fix that problem, Spark needs to be able to rollback a shuffle > map stage and rerun all the map tasks. > According to https://github.com/apache/spark/pull/9214 , Spark doesn't > support it currently, as in shuffle writing "first write wins". > Since overwriting shuffle files is hard, we can extend the shuffle id to > include a "shuffle generation number". Then the reduce task can specify which > generation of shuffle it wants to read. > https://github.com/apache/spark/pull/6648 seems in the right direction. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org