Github user markhamstra commented on a diff in the pull request:
https://github.com/apache/spark/pull/22771#discussion_r228371144
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1364,6 +1385,21 @@ private[spark] class DAGScheduler(
if (job.numFinished == job.numPartitions) {
markStageAsFinished(resultStage)
cleanupStateForJobAndIndependentStages(job)
+ try {
+ // killAllTaskAttempts will fail if a
SchedulerBackend does not implement
+ // killTask.
+ logInfo(s"Job ${job.jobId} is finished. Killing
potential speculative or " +
+ s"zombie tasks for this job")
--- End diff --
Yes, other log messages probably also aren't very good. Maybe what we need
is some additional explanation in the docs somewhere.
The issue that I am having is that if the log messages say that Tasks are
being killed or canceled or whatever, many users will assume that that means
that the Tasks will no longer be running on the Executors. In fact, what it
means is that the DAGScheduler isn't going to try to run them anymore, and that
any previously started Tasks may or may not still be running or continue to run
on the Executors -- it depends on whether the Tasks are interruptible and on
whether the interrupt on cancel configuration is set to true. The log messages
make sense if you understand that subtlety, so we should probably try to
explain it more fully in the docs.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
