Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/246#discussion_r11189913
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -505,6 +505,21 @@ class DAGScheduler(
}
/**
+ * Cancel all jobs associated with a running or scheduled stage.
+ */
+ def cancelStage(stageId: Int) {
--- End diff --
I think adding an event adds just a few lines of code and handles the race
condition discussed offline (where a job submitted after the cancelStage()
function gets called but before the JobCancellation() event gets processed
won't be cancelled). I'd create a StageCancelled event that this function
calls, and then just move the body of the function to inside the DAGScheduler
event loop in the StageCancelled case (and change the cancelJob() function to
handleJobCancellation()).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
