So in your case , the driver is shutting down gracefully , but the executors are not. IS this the problem?
Thanks Deepak On Thu, May 12, 2016 at 11:49 AM, Rakesh H (Marketing Platform-BLR) < rakes...@flipkart.com> wrote: > Yes, it is set to true. > Log of driver : > > 16/05/12 10:18:29 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM > 16/05/12 10:18:29 INFO streaming.StreamingContext: Invoking > stop(stopGracefully=true) from shutdown hook > 16/05/12 10:18:29 INFO scheduler.JobGenerator: Stopping JobGenerator > gracefully > 16/05/12 10:18:29 INFO scheduler.JobGenerator: Waiting for all received > blocks to be consumed for job generation > 16/05/12 10:18:29 INFO scheduler.JobGenerator: Waited for all received blocks > to be consumed for job generation > > Log of executor: > 16/05/12 10:18:29 ERROR executor.CoarseGrainedExecutorBackend: Driver > xx.xx.xx.xx:xxxxx disassociated! Shutting down. > 16/05/12 10:18:29 WARN remote.ReliableDeliverySupervisor: Association with > remote system [xx.xx.xx.xx:xxxxx] has failed, address is now gated for [5000] > ms. Reason: [Disassociated] > 16/05/12 10:18:29 INFO storage.DiskBlockManager: Shutdown hook called > 16/05/12 10:18:29 INFO processors.StreamJobRunner$: VALUE -------------> 204 > //This is value i am logging > 16/05/12 10:18:29 INFO util.ShutdownHookManager: Shutdown hook called > 16/05/12 10:18:29 INFO processors.StreamJobRunner$: VALUE -------------> 205 > 16/05/12 10:18:29 INFO processors.StreamJobRunner$: VALUE -------------> 206 > > > > > > > On Thu, May 12, 2016 at 11:45 AM Deepak Sharma <deepakmc...@gmail.com> > wrote: > >> Hi Rakesh >> Did you tried setting *spark.streaming.stopGracefullyOnShutdown to true *for >> your spark configuration instance? >> If not try this , and let us know if this helps. >> >> Thanks >> Deepak >> >> On Thu, May 12, 2016 at 11:42 AM, Rakesh H (Marketing Platform-BLR) < >> rakes...@flipkart.com> wrote: >> >>> Issue i am having is similar to the one mentioned here : >>> >>> http://stackoverflow.com/questions/36911442/how-to-stop-gracefully-a-spark-streaming-application-on-yarn >>> >>> I am creating a rdd from sequence of 1 to 300 and creating streaming RDD >>> out of it. >>> >>> val rdd = ssc.sparkContext.parallelize(1 to 300) >>> val dstream = new ConstantInputDStream(ssc, rdd) >>> dstream.foreachRDD{ rdd => >>> rdd.foreach{ x => >>> log(x) >>> Thread.sleep(50) >>> } >>> } >>> >>> >>> When i kill this job, i expect elements 1 to 300 to be logged before >>> shutting down. It is indeed the case when i run it locally. It wait for the >>> job to finish before shutting down. >>> >>> But when i launch the job in custer with "yarn-cluster" mode, it >>> abruptly shuts down. >>> Executor prints following log >>> >>> ERROR executor.CoarseGrainedExecutorBackend: >>> Driver xx.xx.xx.xxx:yyyyy disassociated! Shutting down. >>> >>> and then it shuts down. It is not a graceful shutdown. >>> >>> Anybody knows how to do it in yarn ? >>> >>> >>> >>> >> >> >> -- >> Thanks >> Deepak >> www.bigdatabig.com >> www.keosha.net >> > -- Thanks Deepak www.bigdatabig.com www.keosha.net
