[
https://issues.apache.org/jira/browse/SPARK-33490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srimantha Rajapaksa updated SPARK-33490:
Summary: calling system.exit(-1) from a future does not result in the
driver reflecting the fact that the spark app failed. (was: calling
system.exit(-1) from a future does not result in the driver reflecting the fact
that spark app failed.)
> calling system.exit(-1) from a future does not result in the driver
> reflecting the fact that the spark app failed.
> ---
>
> Key: SPARK-33490
> URL: https://issues.apache.org/jira/browse/SPARK-33490
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Srimantha Rajapaksa
>Priority: Major
>
> Hi Guys,
> when calling system. exit(-1) from within a code block defined for a future
> observed that the vm does not exit immediately as you would expect a java app
> to do. in fact it seems to exit the future thread and the diver program
> carries on as normal. the status of the app when it exits is also misleading.
> It says FINISHED instead of FAILED. I have attached code sample see below)
> that replicate this issue. is this a spark bug or a "feature" :) any
> thoughts?
> Note: I have been able to work around this by not calling system .exit(-1)
> from within a future but this was definitely a gotcha that I thought
> worthwhile raising..
> {code:java}
> // code placeholder
> package com.spark.example
> import com.typesafe.config.Config
> import com.typesafe.scalalogging.Logger
> import org.apache.spark.rdd.RDD
> import org.apache.spark.sql.SparkSession
> import scala.concurrent.ExecutionContext.Implicits.global
> import scala.concurrent.duration.DurationInt
> import scala.concurrent.{Await, Future}
> import scala.util.{Failure, Success, Try}
> object SystemExitTestApp extends App {
> val appName = "NumberCountSparkApp"
> val logger = Logger(getClass)
> val sparkMaster = "spark://adl-matspm01:7077"
> val ss = SparkSession.builder()
> .master(sparkMaster)
> .appName(appName).getOrCreate()
> logger.info(s"Starting test. thread id ${Thread.currentThread().getId}")
> val data = Seq(1, 2, 3, 4)
> val rdd: RDD[Int] = ss.sparkContext.parallelize(data)
> val future = Future {
> logger.info(s"inside future thread id ${Thread.currentThread().getId}")
> Try {
> throw new Exception("My dummy exception")
> rdd.count()
> } match {
> case Success(_) => {
> (true, 1)
> }
> case Failure(exception) => {
> logger.info(s"inside future. Exception ${exception} thread id
> ${Thread.currentThread().getId}")
> (false, 0)
> }
> }
> }
> var result: Boolean = true
> future onComplete {
> case Success(x) => {
> logger.info(s"success ${x}")
> if (!x._1) {
> logger.info(s"exiting in the success block of on complete. thread id
> ${Thread.currentThread().getId}")
> result = x._1
> ss.close()
> sys.exit(-1)
> } else {
> logger.info(s"not exiting in the success block of on complete. thread
> id ${Thread.currentThread().getId}")
> result = x._1
> }
> }
> case Failure(exception) => {
> logger.info(s"exiting in the failure block of on complete as exception
> occurred ${exception}. thread id ${Thread.currentThread().getId}")
> result = false
> sys.exit(-1)
> }
> }
> Await.result(future, 3 hours)
> logger.info(s"should not see this really if we do lets use the result
> outside. thread id ${Thread.currentThread().getId}")
> if (!result) {
> logger.info(s" inside main block should not see this really. thread id
> ${Thread.currentThread().getId}")
> //sys.exit(-1)
> }
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
