Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/6291#discussion_r36262816 --- Diff: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- @@ -749,6 +752,43 @@ class DAGSchedulerSuite assertDataStructuresEmpty() } + /** + * Makes sure that tasks for a stage used by multiple jobs are submitted with the properties of a + * later, active job if they were previously run under a job that is no longer active + */ + test("stage used by two jobs, the first no longer active") { + val baseRdd = new MyRDD(sc, 1, Nil) + val finalRdd1 = new MyRDD(sc, 1, List(new OneToOneDependency(baseRdd))) + val finalRdd2 = new MyRDD(sc, 1, List(new OneToOneDependency(baseRdd))) --- End diff -- is the problem that w/ a OnetoOneDependency, there isn't actually a shared stage? I think there will be a shared RDD, but the stages are still separate. I'm thinking you need a test where you have a shared ShuffleDependency.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org