RE: Yarn Shutting Down Spark Processing

2015-09-23 Thread Bryan
a Das" <t...@databricks.com> Sent: ‎9/‎23/‎2015 2:46 AM To: "Bryan Jeffrey" <bryan.jeff...@gmail.com> Cc: "user" <user@spark.apache.org> Subject: Re: Yarn Shutting Down Spark Processing Does your simple Spark batch jobs work in the same YARN setup? May be

Re: Yarn Shutting Down Spark Processing

2015-09-23 Thread Tathagata Das
Does your simple Spark batch jobs work in the same YARN setup? May be YARN is not able to provide resources that you are asking for. On Tue, Sep 22, 2015 at 5:49 PM, Bryan Jeffrey wrote: > Hello. > > I have a Spark streaming job running on a cluster managed by Yarn.

Re: Yarn Shutting Down Spark Processing

2015-09-23 Thread Tathagata Das
CC;ing Hari who may have a better sense of whats going on. -- Forwarded message -- From: Bryan <bryan.jeff...@gmail.com> Date: Wed, Sep 23, 2015 at 3:43 AM Subject: RE: Yarn Shutting Down Spark Processing To: Tathagata Das <t...@databricks.com> Cc: user <user@s

RE: Yarn Shutting Down Spark Processing

2015-09-23 Thread Bryan
;; "Hari Shreedharan" <hshreedha...@cloudera.com> Subject: Re: Yarn Shutting Down Spark Processing CC;ing Hari who may have a better sense of whats going on. -- Forwarded message -- From: Bryan <bryan.jeff...@gmail.com> Date: Wed, Sep 23, 2015

Re: Yarn Shutting Down Spark Processing

2015-09-23 Thread Marcelo Vanzin
Did you look at your application's logs (using the "yarn logs" command?). That error means your application is failing to create a SparkContext. So either you have a bug in your code, or there will be some error in the log pointing at the actual reason for the failure. On Tue, Sep 22, 2015 at

RE: Yarn Shutting Down Spark Processing

2015-09-23 Thread Bryan
-Original Message- From: "Marcelo Vanzin" <van...@cloudera.com> Sent: ‎9/‎23/‎2015 5:55 PM To: "Bryan Jeffrey" <bryan.jeff...@gmail.com> Cc: "user" <user@spark.apache.org> Subject: Re: Yarn Shutting Down Spark Processing Did you look at your a

Re: Yarn Shutting Down Spark Processing

2015-09-23 Thread Marcelo Vanzin
But that's not the complete application log. You say the streaming context is initialized, but can you show that in the logs? There's something happening that is causing the SparkContext to not be registered with the YARN backend, and that's why your application is being killed. If you can share

Yarn Shutting Down Spark Processing

2015-09-22 Thread Bryan Jeffrey
Hello. I have a Spark streaming job running on a cluster managed by Yarn. The spark streaming job starts and receives data from Kafka. It is processing well and then after several seconds I see the following error: 15/09/22 14:53:49 ERROR yarn.ApplicationMaster: SparkContext did not initialize