If you wanted to stop it gracefully, then why are you not calling
ssc.stop(stopGracefully = true, stopSparkContext = true)? Then it doesnt
matter whether the shutdown hook was called or not.
TD
On Mon, May 18, 2015 at 9:43 PM, Dibyendu Bhattacharya <
dibyendu.bhattach...@gmail.com> wrote:
> Hi,
>
> Just figured out that if I want to perform graceful shutdown of Spark
> Streaming 1.4 ( from master ) , the Runtime.getRuntime().addShutdownHook no
> longer works . As in Spark 1.4 there is Utils.addShutdownHook defined for
> Spark Core, that gets anyway called , which leads to graceful shutdown from
> Spark streaming failed with error like "Sparkcontext already closed" issue.
>
> To solve this , I need to explicitly add Utils.addShutdownHook in my
> driver with higher priority ( say 150 ) than Spark's shutdown priority of
> 50 , and there I specified streamingcontext stop method with (false , true)
> parameter.
>
> Just curious to know , if this is how we need to handle shutdown hook
> going forward ?
>
> Can't we make the streaming shutdown default to gracefully shutdown ?
>
> Also the Java Api for adding shutdownhook in Utils looks very dirty with
> methods like this ..
>
>
>
> Utils.addShutdownHook(150, new Function0<BoxedUnit>() {
> @Override
> public BoxedUnit apply() {
> return null;
> }
>
> @Override
> public byte apply$mcB$sp() {
> return 0;
> }
>
> @Override
> public char apply$mcC$sp() {
> return 0;
> }
>
> @Override
> public double apply$mcD$sp() {
> return 0;
> }
>
> @Override
> public float apply$mcF$sp() {
> return 0;
> }
>
> @Override
> public int apply$mcI$sp() {
> // TODO Auto-generated method stub
> return 0;
> }
>
> @Override
> public long apply$mcJ$sp() {
> return 0;
> }
>
> @Override
> public short apply$mcS$sp() {
> return 0;
> }
>
> @Override
> public void apply$mcV$sp() {
> *jsc.stop(false, true);*
> }
>
> @Override
> public boolean apply$mcZ$sp() {
> // TODO Auto-generated method stub
> return false;
> }
> });
>
