Repository: spark Updated Branches: refs/heads/master f28b1a4c4 -> 26e66453d
[SPARK-22655][PYSPARK] Throw exception rather than exit silently in PythonRunner when Spark session is stopped ## What changes were proposed in this pull request? During Spark shutdown, if there are some active tasks, sometimes they will complete with incorrect results. The issue is in PythonRunner where it is returning partial result instead of throwing exception during Spark shutdown. This patch makes it so that these tasks fail instead of complete with partial results. ## How was this patch tested? Existing tests. Author: Li Jin <ice.xell...@gmail.com> Closes #19852 from icexelloss/python-runner-shutdown. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/26e66453 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/26e66453 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/26e66453 Branch: refs/heads/master Commit: 26e66453decf40ed6d590498aadbbf442bb90622 Parents: f28b1a4 Author: Li Jin <ice.xell...@gmail.com> Authored: Fri Dec 8 20:44:21 2017 +0900 Committer: hyukjinkwon <gurwls...@gmail.com> Committed: Fri Dec 8 20:44:21 2017 +0900 ---------------------------------------------------------------------- .../main/scala/org/apache/spark/api/python/PythonRunner.scala | 4 ---- 1 file changed, 4 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/26e66453/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala b/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala index f524de6..93d508c 100644 --- a/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala +++ b/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala @@ -317,10 +317,6 @@ private[spark] abstract class BasePythonRunner[IN, OUT]( logDebug("Exception thrown after task interruption", e) throw new TaskKilledException(context.getKillReason().getOrElse("unknown reason")) - case e: Exception if env.isStopped => - logDebug("Exception thrown after context is stopped", e) - null.asInstanceOf[OUT] // exit silently - case e: Exception if writerThread.exception.isDefined => logError("Python worker exited unexpectedly (crashed)", e) logError("This may have been caused by a prior exception:", writerThread.exception.get) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
