Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22149#discussion_r235266375 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala --- @@ -308,6 +308,12 @@ private class ScriptTransformationWriterThread( } threwException = false } catch { + // TaskKilledException should not be thrown again, otherwise it will be captured by + // SparkUncaughtExceptionHandler, then Executor will exit because of TaskKilledException. + case e: TaskKilledException => --- End diff -- I see. So `ScriptTransformationExec` is special because 1. it starts a new thread 2. the new thread is very likely to throw `TaskKilledException`, when speculation is on. I think we should not kill the executor just because `ScriptTransformationWriterThread` fails. We should log the error, instead of throwing it.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org