Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4008#issuecomment-69657578 To quickly trace through the current behavior: - The ["kill" link handler](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/jobs/StagesTab.scala#L42) calls `SparkContext.cancelStage()`. - [This](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L1436), in turn, calls DAGScheduler.cancelStage(). - In the [handleStageCancellation](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1148), handler, DAGScheduler finds all jobs that depend on a stage and cancels those jobs. - [This](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1165), in turn, [fails](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1194), the job and all stages used only by that job ("independent stages"). When failing a stage, DAGScheduler [tries to cancel](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1221) any tasks running in that stage. For a while, the Mesos scheduler backend didn't support task cancellation, but this has been fixed. Task cancellation is asynchronous and best-effort, but the expected behavior is that canceling a stage will kill any running tasks in that stage. I guess the issue here is that the task cancellation isn't sufficient to clean up the receiver because it's run in a separate thread that's not cleaned up when the task exits. We can't necessarily rely on threads being interrupted when they're cancelled (see https://github.com/apache/spark/commit/432201c7ee9e1ea1d70a6418cbad1c5ad2653ed3), so this seems like a reasonable approach. I have a couple of code-level comments about the approach proposed here, though, so let me comment on those in-line.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org