Github user kayousterhout commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14079#discussion_r92231466
  
    --- Diff: 
core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala 
---
    @@ -157,8 +160,16 @@ abstract class SchedulerIntegrationSuite[T <: 
MockBackend: ClassTag] extends Spa
           }
           // When a job fails, we terminate before waiting for all the task 
end events to come in,
           // so there might still be a running task set.  So we only check 
these conditions
    -      // when the job succeeds
    -      assert(taskScheduler.runningTaskSets.isEmpty)
    +      // when the job succeeds.
    +      // When the final task of a taskset completes, we post
    +      // the event to the DAGScheduler event loop before we finish 
processing in the taskscheduler
    +      // thread.  Its possible the DAGScheduler thread processes the 
event, finishes the job,
    +      // and notifies the job waiter before our original thread in the 
task scheduler finishes
    +      // handling the event and marks the taskset as complete.  So its ok 
if we need to wait a
    +      // *little* bit longer for the original taskscheduler thread to 
finish up to deal w/ the race.
    +      eventually(timeout(1 second), interval(100 millis)) {
    +        assert(taskScheduler.runningTaskSets.isEmpty)
    --- End diff --
    
    Cool sounds good re: separate fix.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to