[ https://issues.apache.org/jira/browse/SPARK-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257481#comment-15257481 ]
Takuya Ueshin commented on SPARK-13902: --------------------------------------- I'm sorry, I made a mistake. I should have written the number in the diagram, but I wrote the actual shuffle id from DAGScheduler we can get when we run the test. I'll update it. So the rest of your questions are right. > Make DAGScheduler.getAncestorShuffleDependencies() return in topological > order to ensure building ancestor stages first. > ------------------------------------------------------------------------------------------------------------------------ > > Key: SPARK-13902 > URL: https://issues.apache.org/jira/browse/SPARK-13902 > Project: Spark > Issue Type: Bug > Components: Scheduler > Reporter: Takuya Ueshin > > {{DAGScheduler}} sometimes generate incorrect stage graph. > Some stages are generated for the same shuffleId twice or more and they are > referenced by the child stages because the building order of the graph is not > correct. > Here, we submit an RDD\[F\] having a linage of RDDs as follows (please see > this in {{monospaced}} font): > {noformat} > <-------------------- > / \ > [A] <--(1)-- [B] <--(2)-- [C] <--(3)-- [D] <--(4)-- [E] <--(5)-- [F] > \ / > <-------------------- > {noformat} > {{DAGScheduler}} generates the following stages and their parents for each > shuffle id: > | shuffle id | stage | parents | > | 0 | ShuffleMapStage 2 | List() | > | 1 | ShuffleMapStage 1 | List(ShuffleMapStage 0) | > | 2 | ShuffleMapStage 3 | List(ShuffleMapStage 1) | > | 3 | ShuffleMapStage 4 | List(ShuffleMapStage 2, ShuffleMapStage 3) | > | 4 | ShuffleMapStage 5 | List(ShuffleMapStage 1, ShuffleMapStage 4) | > | \- | ResultStage 6 | List(ShuffleMapStage 5) | > The stage for shuffle id {{0}} should be {{ShuffleMapStage 0}}, but the stage > for shuffle id {{0}} is generated twice as {{ShuffleMapStage 2}} and > {{ShuffleMapStage 0}} is overwritten by {{ShuffleMapStage 2}}, and the stage > {{ShuffleMap Stage1}} keeps referring the _old_ stage {{ShuffleMapStage 0}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org