pache.spark.rdd.RDD.iterator(RDD.scala:264)
>
> at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:88)
>
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scal
or.default-dispatcher-4]:
Sent stop signal to all 42 receivers
发件人: Shixiong(Ryan) Zhu [mailto:shixi...@databricks.com]
发送时间: 2016年1月16日 6:28
收件人: 邓刚[技术中心]
抄送: Yogesh Mahajan; user
主题: Re: 答复: 答复: 答复: spark streaming context trigger invoke stop why?
I see. So when your job fails, `jsc.awaitTerm
gt;
> jsc.awaitTermination();
>
>
>
>
>
> *发件人:* Shixiong(Ryan) Zhu [mailto:shixi...@databricks.com]
> *发送时间:* 2016年1月15日 6:02
> *收件人:* 邓刚[技术中心]
> *抄送:* Yogesh Mahajan; user
> *主题:* Re: 答复: 答复: spark streaming context trigger invoke stop why?
>
>
>
nes.d...@vipshop.com>> wrote:
What’s more, I am running a 7*24 hours job , so I won’t call System.exit() by
myself. So I believe somewhere of the driver kill itself
发件人: 邓刚[技术中心]
发送时间: 2016年1月14日 15:45
收件人: 'Yogesh Mahajan'
抄送: user
主题: 答复: 答复: spark streaming context trigger invoke s
stem.exit()
> by myself. So I believe somewhere of the driver kill itself
>
>
>
> *发件人:* 邓刚[技术中心]
> *发送时间:* 2016年1月14日 15:45
> *收件人:* 'Yogesh Mahajan'
> *抄送:* user
> *主题:* 答复: 答复: spark streaming context trigger invoke stop why?
>
>
>
> Thanks for your response, Appli
What’s more, I am running a 7*24 hours job , so I won’t call System.exit() by
myself. So I believe somewhere of the driver kill itself
发件人: 邓刚[技术中心]
发送时间: 2016年1月14日 15:45
收件人: 'Yogesh Mahajan'
抄送: user
主题: 答复: 答复: spark streaming context trigger invoke stop why?
Thanks for your response
context trigger invoke stop why?
All the action happens in ApplicationMaster expecially in run method
Check ApplicationMaster#startUserApplication : userThread(Driver) which invokes
ApplicationMaster#finish method. You can also try System.exit in your program
Regards,
Yogesh Mahajan,
SnappyData Inc
Hi Triones,
Check the org.apache.spark.util.ShutdownHookManager : It adds this
ShutDownHook when you start a StreamingContext
Here is the code in StreamingContext.start()
shutdownHookRef = ShutdownHookManager.addShutdownHook(
StreamingContext.SHUTDOWN_HOOK_PRIORITY)(stopOnShutdown)
More info
I am using spark version 1.5.2
发件人: Triones,Deng(vip.com) [mailto:triones.d...@vipshop.com]
发送时间: 2016年1月14日 11:24
收件人: user
主题: spark streaming context trigger invoke stop why?
Hi all
As I saw the driver log, the task failed 4 times in a stage, the stage
will be dropped
All the action happens in ApplicationMaster expecially in run method
Check ApplicationMaster#startUserApplication : userThread(Driver) which
invokes ApplicationMaster#finish method. You can also try System.exit in
your program
Regards,
Yogesh Mahajan,
SnappyData Inc, snappydata.io
On Thu, Jan
10 matches
Mail list logo