Re: spark-submit stuck and no output in console

[Kayode Odeyemi]

Sonal, SparkPi couldn't run as well. Stuck to the screen with no output

hadoop-user@yks-hadoop-m01:/usr/local/spark$ ./bin/run-example SparkPi

On Tue, Nov 17, 2015 at 12:22 PM, Steve Loughran 
wrote:

> 48 hours is one of those kerberos warning times (as is 24h, 72h and 7
> days) 


Does this mean I need to restart the whole Hadoop YARN cluster to reset
kerberos?




-- 
Odeyemi 'Kayode O.
http://ng.linkedin.com/in/kayodeodeyemi. t: @charyorde

Re: spark-submit stuck and no output in console

[Kayode Odeyemi]

Our hadoop NFS Gateway seems to be malfunctioning.

I basically restart it. Now spark jobs have resumed successfully.

Problem solved.

Re: spark-submit stuck and no output in console

[Sonal Goyal]

I would suggest a couple of things to try

A. Try running the example program with master as local[*]. See if spark
can run locally or not.
B. Check spark master and worker logs.
C. Check if normal hadoop jobs can be run properly on the cluster.
D. Check spark master webui and see health of cluster.
On Nov 17, 2015 4:16 PM, "Kayode Odeyemi"  wrote:

> Sonal, SparkPi couldn't run as well. Stuck to the screen with no output
>
> hadoop-user@yks-hadoop-m01:/usr/local/spark$ ./bin/run-example SparkPi
>
> On Tue, Nov 17, 2015 at 12:22 PM, Steve Loughran 
> wrote:
>
>> 48 hours is one of those kerberos warning times (as is 24h, 72h and 7
>> days) 
>
>
> Does this mean I need to restart the whole Hadoop YARN cluster to reset
> kerberos?
>
>
>
>
> --
> Odeyemi 'Kayode O.
> http://ng.linkedin.com/in/kayodeodeyemi. t: @charyorde
>

Re: spark-submit stuck and no output in console

[Kayode Odeyemi]

Anyone experienced this issue as well?

On Mon, Nov 16, 2015 at 8:06 PM, Kayode Odeyemi  wrote:

>
> Or are you saying that the Java process never even starts?
>
>
> Exactly.
>
> Here's what I got back from jstack as expected:
>
> hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ jstack 31316
> 31316: Unable to open socket file: target process not responding or
> HotSpot VM not loaded
> The -F option can be used when the target process is not responding
> hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ jstack 31316 -F
> Attaching to core -F from executable 31316, please wait...
> Error attaching to core file: Can't attach to the core file
>
>
>


-- 
Odeyemi 'Kayode O.
http://ng.linkedin.com/in/kayodeodeyemi. t: @charyorde

Re: spark-submit stuck and no output in console

[Sonal Goyal]

How did the example spark jobs go? SparkPI etc..?

Best Regards,
Sonal
Founder, Nube Technologies 
Reifier at Strata Hadoop World

Reifier at Spark Summit 2015






On Tue, Nov 17, 2015 at 3:24 PM, Kayode Odeyemi  wrote:

> Thanks for the reply Sonal.
>
> I'm on JDK 7 (/usr/lib/jvm/java-7-oracle)
>
> My env is a YARN cluster made of 7 nodes (6 datanodes/
> node manager, 1 namenode/resource manager).
>
> On the namenode, is where I executed the spark-submit job while on one of
> the datanodes,  I executed 'hadoop fs -put /binstore /user/hadoop-user/'
> to dump 1TB of data into all the datanodes. That process is still running
> without hassle and it's only using 1.3 GB of 1.7g heap space.
>
> Initially, I submitted 2 jobs to the YARN cluster which was running for 2
> days and suddenly stops. Nothing in the logs shows the root cause.
>
>
> On Tue, Nov 17, 2015 at 11:42 AM, Sonal Goyal 
> wrote:
>
>> Could it be jdk related ? Which version are you on?
>>
>> Best Regards,
>> Sonal
>> Founder, Nube Technologies 
>> Reifier at Strata Hadoop World
>> 
>> Reifier at Spark Summit 2015
>> 
>>
>> 
>>
>>
>>
>> On Tue, Nov 17, 2015 at 2:48 PM, Kayode Odeyemi 
>> wrote:
>>
>>> Anyone experienced this issue as well?
>>>
>>> On Mon, Nov 16, 2015 at 8:06 PM, Kayode Odeyemi 
>>> wrote:
>>>

 Or are you saying that the Java process never even starts?


 Exactly.

 Here's what I got back from jstack as expected:

 hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ jstack 31316
 31316: Unable to open socket file: target process not responding or
 HotSpot VM not loaded
 The -F option can be used when the target process is not responding
 hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ jstack 31316 -F
 Attaching to core -F from executable 31316, please wait...
 Error attaching to core file: Can't attach to the core file



>>>
>>>
>>> --
>>> Odeyemi 'Kayode O.
>>> http://ng.linkedin.com/in/kayodeodeyemi. t: @charyorde
>>>
>>
>>
>
>
> --
> Odeyemi 'Kayode O.
> http://ng.linkedin.com/in/kayodeodeyemi. t: @charyorde
>

Re: spark-submit stuck and no output in console

[Kayode Odeyemi]

Thanks for the reply Sonal.

I'm on JDK 7 (/usr/lib/jvm/java-7-oracle)

My env is a YARN cluster made of 7 nodes (6 datanodes/
node manager, 1 namenode/resource manager).

On the namenode, is where I executed the spark-submit job while on one of
the datanodes,  I executed 'hadoop fs -put /binstore /user/hadoop-user/' to
dump 1TB of data into all the datanodes. That process is still running
without hassle and it's only using 1.3 GB of 1.7g heap space.

Initially, I submitted 2 jobs to the YARN cluster which was running for 2
days and suddenly stops. Nothing in the logs shows the root cause.


On Tue, Nov 17, 2015 at 11:42 AM, Sonal Goyal  wrote:

> Could it be jdk related ? Which version are you on?
>
> Best Regards,
> Sonal
> Founder, Nube Technologies 
> Reifier at Strata Hadoop World
> 
> Reifier at Spark Summit 2015
> 
>
> 
>
>
>
> On Tue, Nov 17, 2015 at 2:48 PM, Kayode Odeyemi  wrote:
>
>> Anyone experienced this issue as well?
>>
>> On Mon, Nov 16, 2015 at 8:06 PM, Kayode Odeyemi 
>> wrote:
>>
>>>
>>> Or are you saying that the Java process never even starts?
>>>
>>>
>>> Exactly.
>>>
>>> Here's what I got back from jstack as expected:
>>>
>>> hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ jstack 31316
>>> 31316: Unable to open socket file: target process not responding or
>>> HotSpot VM not loaded
>>> The -F option can be used when the target process is not responding
>>> hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ jstack 31316 -F
>>> Attaching to core -F from executable 31316, please wait...
>>> Error attaching to core file: Can't attach to the core file
>>>
>>>
>>>
>>
>>
>> --
>> Odeyemi 'Kayode O.
>> http://ng.linkedin.com/in/kayodeodeyemi. t: @charyorde
>>
>
>


-- 
Odeyemi 'Kayode O.
http://ng.linkedin.com/in/kayodeodeyemi. t: @charyorde

Re: spark-submit stuck and no output in console

[Sonal Goyal]

Could it be jdk related ? Which version are you on?

Best Regards,
Sonal
Founder, Nube Technologies 
Reifier at Strata Hadoop World

Reifier at Spark Summit 2015






On Tue, Nov 17, 2015 at 2:48 PM, Kayode Odeyemi  wrote:

> Anyone experienced this issue as well?
>
> On Mon, Nov 16, 2015 at 8:06 PM, Kayode Odeyemi  wrote:
>
>>
>> Or are you saying that the Java process never even starts?
>>
>>
>> Exactly.
>>
>> Here's what I got back from jstack as expected:
>>
>> hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ jstack 31316
>> 31316: Unable to open socket file: target process not responding or
>> HotSpot VM not loaded
>> The -F option can be used when the target process is not responding
>> hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ jstack 31316 -F
>> Attaching to core -F from executable 31316, please wait...
>> Error attaching to core file: Can't attach to the core file
>>
>>
>>
>
>
> --
> Odeyemi 'Kayode O.
> http://ng.linkedin.com/in/kayodeodeyemi. t: @charyorde
>

Re: spark-submit stuck and no output in console

[Steve Loughran]

On 17 Nov 2015, at 09:54, Kayode Odeyemi 
> wrote:

Initially, I submitted 2 jobs to the YARN cluster which was running for 2 days 
and suddenly stops. Nothing in the logs shows the root cause.

48 hours is one of those kerberos warning times (as is 24h, 72h and 7 days) 

Re: spark-submit stuck and no output in console

[Kayode Odeyemi]

Spark 1.5.1

The fact is that there's no stack trace. No output from that command at all
to the console.

This is all I get:

hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ tail -1
/tmp/spark-profile-job.log
nohup: ignoring input
/usr/local/spark/bin/spark-class: line 76: 29516 Killed
 "$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"


On Mon, Nov 16, 2015 at 5:22 PM, Ted Yu  wrote:

> Which release of Spark are you using ?
>
> Can you take stack trace and pastebin it ?
>
> Thanks
>
> On Mon, Nov 16, 2015 at 5:50 AM, Kayode Odeyemi  wrote:
>
>> ./spark-submit --class com.migration.UpdateProfiles --executor-memory 8g
>> ~/migration-profiles-0.1-SNAPSHOT.jar
>>
>> is stuck and outputs nothing to the console.
>>
>> What could be the cause of this? Current max heap size is 1.75g and it's
>> only using 1g.
>>
>>
>


-- 
Odeyemi 'Kayode O.
http://ng.linkedin.com/in/kayodeodeyemi. t: @charyorde

Re: spark-submit stuck and no output in console

[Ted Yu]

Which release of Spark are you using ?

Can you take stack trace and pastebin it ?

Thanks

On Mon, Nov 16, 2015 at 5:50 AM, Kayode Odeyemi  wrote:

> ./spark-submit --class com.migration.UpdateProfiles --executor-memory 8g
> ~/migration-profiles-0.1-SNAPSHOT.jar
>
> is stuck and outputs nothing to the console.
>
> What could be the cause of this? Current max heap size is 1.75g and it's
> only using 1g.
>
>

Re: spark-submit stuck and no output in console

[Jonathan Kelly]

He means for you to use jstack to obtain a stacktrace of all of the
threads. Or are you saying that the Java process never even starts?

On Mon, Nov 16, 2015 at 7:48 AM, Kayode Odeyemi  wrote:

> Spark 1.5.1
>
> The fact is that there's no stack trace. No output from that command at
> all to the console.
>
> This is all I get:
>
> hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ tail -1
> /tmp/spark-profile-job.log
> nohup: ignoring input
> /usr/local/spark/bin/spark-class: line 76: 29516 Killed
>  "$RUNNER" -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"
>
>
> On Mon, Nov 16, 2015 at 5:22 PM, Ted Yu  wrote:
>
>> Which release of Spark are you using ?
>>
>> Can you take stack trace and pastebin it ?
>>
>> Thanks
>>
>> On Mon, Nov 16, 2015 at 5:50 AM, Kayode Odeyemi 
>> wrote:
>>
>>> ./spark-submit --class com.migration.UpdateProfiles --executor-memory 8g
>>> ~/migration-profiles-0.1-SNAPSHOT.jar
>>>
>>> is stuck and outputs nothing to the console.
>>>
>>> What could be the cause of this? Current max heap size is 1.75g and it's
>>> only using 1g.
>>>
>>>
>>
>
>
> --
> Odeyemi 'Kayode O.
> http://ng.linkedin.com/in/kayodeodeyemi. t: @charyorde
>

spark-submit stuck and no output in console

[Kayode Odeyemi]

./spark-submit --class com.migration.UpdateProfiles --executor-memory 8g
~/migration-profiles-0.1-SNAPSHOT.jar

is stuck and outputs nothing to the console.

What could be the cause of this? Current max heap size is 1.75g and it's
only using 1g.

Re: spark-submit stuck and no output in console

[Kayode Odeyemi]

> Or are you saying that the Java process never even starts?


Exactly.

Here's what I got back from jstack as expected:

hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ jstack 31316
31316: Unable to open socket file: target process not responding or HotSpot
VM not loaded
The -F option can be used when the target process is not responding
hadoop-user@yks-hadoop-m01:/usr/local/spark/bin$ jstack 31316 -F
Attaching to core -F from executable 31316, please wait...
Error attaching to core file: Can't attach to the core file