Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r92231466
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala
---
@@ -157,8 +160,16 @@ abstract class SchedulerIntegrationSuite[T <:
MockBackend: ClassTag] extends Spa
}
// When a job fails, we terminate before waiting for all the task
end events to come in,
// so there might still be a running task set. So we only check
these conditions
- // when the job succeeds
- assert(taskScheduler.runningTaskSets.isEmpty)
+ // when the job succeeds.
+ // When the final task of a taskset completes, we post
+ // the event to the DAGScheduler event loop before we finish
processing in the taskscheduler
+ // thread. Its possible the DAGScheduler thread processes the
event, finishes the job,
+ // and notifies the job waiter before our original thread in the
task scheduler finishes
+ // handling the event and marks the taskset as complete. So its ok
if we need to wait a
+ // *little* bit longer for the original taskscheduler thread to
finish up to deal w/ the race.
+ eventually(timeout(1 second), interval(100 millis)) {
+ assert(taskScheduler.runningTaskSets.isEmpty)
--- End diff --
Cool sounds good re: separate fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
