[
https://issues.apache.org/jira/browse/SPARK-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307387#comment-15307387
]
Saisai Shao commented on SPARK-15479:
-------------------------------------
I guess using "yarn kill" to stop the application has a different behavior
compared to local mode you run.
If "yarn kill" only kills the AM/Driver, let driver to stop executor, then you
could get the grace shutdown result.
If "yarn kill" just kill all the container belongs to the app immediately, then
grace shutdown is not worked.
So it really depends on the behavior of "yarn kill" and hard to do in the Spark
side.
> Spark job doesn't shut gracefully in yarn mode.
> -----------------------------------------------
>
> Key: SPARK-15479
> URL: https://issues.apache.org/jira/browse/SPARK-15479
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.5.1
> Reporter: Rakesh
> Attachments: driver.rtf, executor.rtf
>
>
> Issue i am having is similar to the one mentioned here :
> http://stackoverflow.com/questions/36911442/how-to-stop-gracefully-a-spark-streaming-application-on-yarn
> I am creating a rdd from sequence of 1 to 300 and creating streaming RDD out
> of it.
> val rdd = ssc.sparkContext.parallelize(1 to 300)
> val dstream = new ConstantInputDStream(ssc, rdd)
> dstream.foreachRDD{ rdd =>
> rdd.foreach{ x =>
> log(x)
> Thread.sleep(50)
> }
> }
> When i kill this job, i expect elements 1 to 300 to be logged before shutting
> down. It is indeed the case when i run it locally. It waits for the job to
> finish before shutting down.
> But when i launch the job in cluster with "yarn-cluster" mode, it abruptly
> shuts down.
> Executor prints following log
> ERROR executor.CoarseGrainedExecutorBackend:
> Driver xx.xx.xx.xxx:yyyyy disassociated! Shutting down.
> and then it shuts down. It is not a graceful shutdown.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
