Re: 答复: 答复: 答复: 答复: spark streaming context trigger invoke stop why?

2016-01-18 Thread Shixiong(Ryan) Zhu
pache.spark.rdd.RDD.iterator(RDD.scala:264) > > at > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) > > at org.apache.spark.scheduler.Task.run(Task.scala:88) > > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scal

答复: 答复: 答复: 答复: spark streaming context trigger invoke stop why?

2016-01-16 Thread Triones,Deng(vip.com)
or.default-dispatcher-4]: Sent stop signal to all 42 receivers 发件人: Shixiong(Ryan) Zhu [mailto:shixi...@databricks.com] 发送时间: 2016年1月16日 6:28 收件人: 邓刚[技术中心] 抄送: Yogesh Mahajan; user 主题: Re: 答复: 答复: 答复: spark streaming context trigger invoke stop why? I see. So when your job fails, `jsc.awaitTerm

Re: 答复: 答复: 答复: spark streaming context trigger invoke stop why?

2016-01-15 Thread Shixiong(Ryan) Zhu
gt; > jsc.awaitTermination(); > > > > > > *发件人:* Shixiong(Ryan) Zhu [mailto:shixi...@databricks.com] > *发送时间:* 2016年1月15日 6:02 > *收件人:* 邓刚[技术中心] > *抄送:* Yogesh Mahajan; user > *主题:* Re: 答复: 答复: spark streaming context trigger invoke stop why? > > >

答复: 答复: 答复: spark streaming context trigger invoke stop why?

2016-01-14 Thread Triones,Deng(vip.com)
nes.d...@vipshop.com>> wrote: What’s more, I am running a 7*24 hours job , so I won’t call System.exit() by myself. So I believe somewhere of the driver kill itself 发件人: 邓刚[技术中心] 发送时间: 2016年1月14日 15:45 收件人: 'Yogesh Mahajan' 抄送: user 主题: 答复: 答复: spark streaming context trigger invoke s

Re: 答复: 答复: spark streaming context trigger invoke stop why?

2016-01-14 Thread Shixiong(Ryan) Zhu
stem.exit() > by myself. So I believe somewhere of the driver kill itself > > > > *发件人:* 邓刚[技术中心] > *发送时间:* 2016年1月14日 15:45 > *收件人:* 'Yogesh Mahajan' > *抄送:* user > *主题:* 答复: 答复: spark streaming context trigger invoke stop why? > > > > Thanks for your response, Appli

答复: 答复: spark streaming context trigger invoke stop why?

2016-01-13 Thread Triones,Deng(vip.com)
What’s more, I am running a 7*24 hours job , so I won’t call System.exit() by myself. So I believe somewhere of the driver kill itself 发件人: 邓刚[技术中心] 发送时间: 2016年1月14日 15:45 收件人: 'Yogesh Mahajan' 抄送: user 主题: 答复: 答复: spark streaming context trigger invoke stop why? Thanks for your response

答复: 答复: spark streaming context trigger invoke stop why?

2016-01-13 Thread Triones,Deng(vip.com)
context trigger invoke stop why? All the action happens in ApplicationMaster expecially in run method Check ApplicationMaster#startUserApplication : userThread(Driver) which invokes ApplicationMaster#finish method. You can also try System.exit in your program Regards, Yogesh Mahajan, SnappyData Inc

Re: 答复: spark streaming context trigger invoke stop why?

2016-01-13 Thread Yogesh Mahajan
Hi Triones, Check the org.apache.spark.util.ShutdownHookManager : It adds this ShutDownHook when you start a StreamingContext Here is the code in StreamingContext.start() shutdownHookRef = ShutdownHookManager.addShutdownHook( StreamingContext.SHUTDOWN_HOOK_PRIORITY)(stopOnShutdown)

答复: spark streaming context trigger invoke stop why?

2016-01-13 Thread Triones,Deng(vip.com)
More info I am using spark version 1.5.2 发件人: Triones,Deng(vip.com) [mailto:triones.d...@vipshop.com] 发送时间: 2016年1月14日 11:24 收件人: user 主题: spark streaming context trigger invoke stop why? Hi all As I saw the driver log, the task failed 4 times in a stage, the stage will be dropped

Re: 答复: spark streaming context trigger invoke stop why?

2016-01-13 Thread Yogesh Mahajan
All the action happens in ApplicationMaster expecially in run method Check ApplicationMaster#startUserApplication : userThread(Driver) which invokes ApplicationMaster#finish method. You can also try System.exit in your program Regards, Yogesh Mahajan, SnappyData Inc, snappydata.io On Thu, Jan