Dose spark auto invoke StreamingContext.stop while receive kill signal?

[Bin Wang]

I'd like the spark application to be stopped gracefully while received kill
signal, so I add these code:

sys.ShutdownHookThread {
  println("Gracefully stopping Spark Streaming Application")
  ssc.stop(stopSparkContext = true, stopGracefully = true)
  println("Application stopped")
}

But the application is not stopped gracefully:

15/09/23 17:44:38 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
...
15/09/23 17:44:38 INFO streaming.StreamingContext: Invoking
stop(stopGracefully=false) from shutdown hook

Dose spark auto invoke StreamingContext.stop for me?

Re: Dose spark auto invoke StreamingContext.stop while receive kill signal?

[Tathagata Das]

YEs, since 1.4.0, it shuts down streamingContext without gracefully from
shutdown hook.
You can make it shutdown gracefully in that hook by setting the SparkConf
"spark.streaming.stopGracefullyOnShutdown" to "true"

Note to self, document this in the programming guide.

On Wed, Sep 23, 2015 at 3:33 AM, Bin Wang  wrote:

> I'd like the spark application to be stopped gracefully while received
> kill signal, so I add these code:
>
> sys.ShutdownHookThread {
>   println("Gracefully stopping Spark Streaming Application")
>   ssc.stop(stopSparkContext = true, stopGracefully = true)
>   println("Application stopped")
> }
>
> But the application is not stopped gracefully:
>
> 15/09/23 17:44:38 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
> ...
> 15/09/23 17:44:38 INFO streaming.StreamingContext: Invoking
> stop(stopGracefully=false) from shutdown hook
>
> Dose spark auto invoke StreamingContext.stop for me?
>