Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r92229579
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala
---
@@ -157,8 +160,16 @@ abstract class SchedulerIntegrationSuite[T <:
MockBackend: ClassTag] extends Spa
}
// When a job fails, we terminate before waiting for all the task
end events to come in,
// so there might still be a running task set. So we only check
these conditions
- // when the job succeeds
- assert(taskScheduler.runningTaskSets.isEmpty)
+ // when the job succeeds.
+ // When the final task of a taskset completes, we post
+ // the event to the DAGScheduler event loop before we finish
processing in the taskscheduler
+ // thread. Its possible the DAGScheduler thread processes the
event, finishes the job,
+ // and notifies the job waiter before our original thread in the
task scheduler finishes
+ // handling the event and marks the taskset as complete. So its ok
if we need to wait a
+ // *little* bit longer for the original taskscheduler thread to
finish up to deal w/ the race.
+ eventually(timeout(1 second), interval(100 millis)) {
+ assert(taskScheduler.runningTaskSets.isEmpty)
--- End diff --
no, just that one of test runs failed because of this
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68694/consoleFull
so I included the fix here in case there was more flakiness. But actually
I dont' think I've seen this failure any other time, so I can move it out to a
separate fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
