Have you used awaitTermination() on your ssc ? --> Yes, i have used that. Also try setting the deployment mode to yarn-client. --> Is this not supported on yarn-cluster mode? I am trying to find root cause for yarn-cluster mode. Have you tested graceful shutdown on yarn-cluster mode?
On Fri, May 13, 2016 at 11:54 AM Deepak Sharma <deepakmc...@gmail.com> wrote: > Rakesh > Have you used awaitTermination() on your ssc ? > If not , dd this and see if it changes the behavior. > I am guessing this issue may be related to yarn deployment mode. > Also try setting the deployment mode to yarn-client. > > Thanks > Deepak > > > On Fri, May 13, 2016 at 10:17 AM, Rakesh H (Marketing Platform-BLR) < > rakes...@flipkart.com> wrote: > >> Ping!! >> Has anybody tested graceful shutdown of a spark streaming in yarn-cluster >> mode?It looks like a defect to me. >> >> >> On Thu, May 12, 2016 at 12:53 PM Rakesh H (Marketing Platform-BLR) < >> rakes...@flipkart.com> wrote: >> >>> We are on spark 1.5.1 >>> Above change was to add a shutdown hook. >>> I am not adding shutdown hook in code, so inbuilt shutdown hook is being >>> called. >>> Driver signals that it is going to to graceful shutdown, but executor >>> sees that Driver is dead and it shuts down abruptly. >>> Could this issue be related to yarn? I see correct behavior locally. I >>> did "yarn kill ...." to kill the job. >>> >>> >>> On Thu, May 12, 2016 at 12:28 PM Deepak Sharma <deepakmc...@gmail.com> >>> wrote: >>> >>>> This is happening because spark context shuts down without shutting >>>> down the ssc first. >>>> This was behavior till spark 1.4 ans was addressed in later releases. >>>> https://github.com/apache/spark/pull/6307 >>>> >>>> Which version of spark are you on? >>>> >>>> Thanks >>>> Deepak >>>> >>>> On Thu, May 12, 2016 at 12:14 PM, Rakesh H (Marketing Platform-BLR) < >>>> rakes...@flipkart.com> wrote: >>>> >>>>> Yes, it seems to be the case. >>>>> In this case executors should have continued logging values till 300, >>>>> but they are shutdown as soon as i do "yarn kill ......" >>>>> >>>>> On Thu, May 12, 2016 at 12:11 PM Deepak Sharma <deepakmc...@gmail.com> >>>>> wrote: >>>>> >>>>>> So in your case , the driver is shutting down gracefully , but the >>>>>> executors are not. >>>>>> IS this the problem? >>>>>> >>>>>> Thanks >>>>>> Deepak >>>>>> >>>>>> On Thu, May 12, 2016 at 11:49 AM, Rakesh H (Marketing Platform-BLR) < >>>>>> rakes...@flipkart.com> wrote: >>>>>> >>>>>>> Yes, it is set to true. >>>>>>> Log of driver : >>>>>>> >>>>>>> 16/05/12 10:18:29 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: >>>>>>> SIGTERM >>>>>>> 16/05/12 10:18:29 INFO streaming.StreamingContext: Invoking >>>>>>> stop(stopGracefully=true) from shutdown hook >>>>>>> 16/05/12 10:18:29 INFO scheduler.JobGenerator: Stopping JobGenerator >>>>>>> gracefully >>>>>>> 16/05/12 10:18:29 INFO scheduler.JobGenerator: Waiting for all received >>>>>>> blocks to be consumed for job generation >>>>>>> 16/05/12 10:18:29 INFO scheduler.JobGenerator: Waited for all received >>>>>>> blocks to be consumed for job generation >>>>>>> >>>>>>> Log of executor: >>>>>>> 16/05/12 10:18:29 ERROR executor.CoarseGrainedExecutorBackend: Driver >>>>>>> xx.xx.xx.xx:xxxxx disassociated! Shutting down. >>>>>>> 16/05/12 10:18:29 WARN remote.ReliableDeliverySupervisor: Association >>>>>>> with remote system [xx.xx.xx.xx:xxxxx] has failed, address is now gated >>>>>>> for [5000] ms. Reason: [Disassociated] >>>>>>> 16/05/12 10:18:29 INFO storage.DiskBlockManager: Shutdown hook called >>>>>>> 16/05/12 10:18:29 INFO processors.StreamJobRunner$: VALUE >>>>>>> -------------> 204 //This is value i am logging >>>>>>> 16/05/12 10:18:29 INFO util.ShutdownHookManager: Shutdown hook called >>>>>>> 16/05/12 10:18:29 INFO processors.StreamJobRunner$: VALUE >>>>>>> -------------> 205 >>>>>>> 16/05/12 10:18:29 INFO processors.StreamJobRunner$: VALUE >>>>>>> -------------> 206 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, May 12, 2016 at 11:45 AM Deepak Sharma < >>>>>>> deepakmc...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Rakesh >>>>>>>> Did you tried setting *spark.streaming.stopGracefullyOnShutdown to >>>>>>>> true *for your spark configuration instance? >>>>>>>> If not try this , and let us know if this helps. >>>>>>>> >>>>>>>> Thanks >>>>>>>> Deepak >>>>>>>> >>>>>>>> On Thu, May 12, 2016 at 11:42 AM, Rakesh H (Marketing Platform-BLR) >>>>>>>> <rakes...@flipkart.com> wrote: >>>>>>>> >>>>>>>>> Issue i am having is similar to the one mentioned here : >>>>>>>>> >>>>>>>>> http://stackoverflow.com/questions/36911442/how-to-stop-gracefully-a-spark-streaming-application-on-yarn >>>>>>>>> >>>>>>>>> I am creating a rdd from sequence of 1 to 300 and creating >>>>>>>>> streaming RDD out of it. >>>>>>>>> >>>>>>>>> val rdd = ssc.sparkContext.parallelize(1 to 300) >>>>>>>>> val dstream = new ConstantInputDStream(ssc, rdd) >>>>>>>>> dstream.foreachRDD{ rdd => >>>>>>>>> rdd.foreach{ x => >>>>>>>>> log(x) >>>>>>>>> Thread.sleep(50) >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> When i kill this job, i expect elements 1 to 300 to be logged >>>>>>>>> before shutting down. It is indeed the case when i run it locally. It >>>>>>>>> wait >>>>>>>>> for the job to finish before shutting down. >>>>>>>>> >>>>>>>>> But when i launch the job in custer with "yarn-cluster" mode, it >>>>>>>>> abruptly shuts down. >>>>>>>>> Executor prints following log >>>>>>>>> >>>>>>>>> ERROR executor.CoarseGrainedExecutorBackend: >>>>>>>>> Driver xx.xx.xx.xxx:yyyyy disassociated! Shutting down. >>>>>>>>> >>>>>>>>> and then it shuts down. It is not a graceful shutdown. >>>>>>>>> >>>>>>>>> Anybody knows how to do it in yarn ? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Thanks >>>>>>>> Deepak >>>>>>>> www.bigdatabig.com >>>>>>>> www.keosha.net >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks >>>>>> Deepak >>>>>> www.bigdatabig.com >>>>>> www.keosha.net >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks >>>> Deepak >>>> www.bigdatabig.com >>>> www.keosha.net >>>> >>> > > > -- > Thanks > Deepak > www.bigdatabig.com > www.keosha.net >
