[jira] [Updated] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later

2014-11-06 Thread Tsuyoshi OZAWA (JIRA) 

 [ 
https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi OZAWA updated SPARK-4267:
--
Description: 
Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5 uses 
protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this:

{code}
 ./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn -Dhadoop.version=2.5.1 
-Dprotobuf.version=2.5.0
{code}

Then Spark on YARN cannot fail to run with NPE.

{code}
$ bin/spark-shell --master yarn-client
scala> sc.textFile("hdfs:///user/ozawa/wordcountInput20G").flatMap(line => 
line.split(" ")).map(word => (word, 1)).persist().reduceByKey((a, b) => a + b, 
16).saveAsTextFile("hdfs:///user/ozawa/sparkWordcountOutNew2");
java.lang.NullPointerException  


  
at 
org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284)
at 
org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291) 


   
at 
org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480)
at $iwC$$iwC$$iwC$$iwC.(:13) 


  
at $iwC$$iwC$$iwC.(:18)
at $iwC$$iwC.(:20)   


  
at $iwC.(:22)
at (:24) 


  
at .(:28)
at .() 


  
at .(:7)
at .() 


  
at $print()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  


  
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)   


   
at 
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
at 
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)  


   
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610) 


  
at 
org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:823)
at 
org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:868)


   
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:780)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:625)

[jira] [Updated] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later

2014-11-06 Thread Tsuyoshi OZAWA (JIRA) 

 [ 
https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tsuyoshi OZAWA updated SPARK-4267:
--
Description: 
Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5 uses 
protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this:

{code}
 ./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn -Dhadoop.version=2.5.1 
-Dprotobuf.version=2.5.0
{code}

Then Spark on YARN fails to launch jobs with NPE.

{code}
$ bin/spark-shell --master yarn-client
scala> sc.textFile("hdfs:///user/ozawa/wordcountInput20G").flatMap(line => 
line.split(" ")).map(word => (word, 1)).persist().reduceByKey((a, b) => a + b, 
16).saveAsTextFile("hdfs:///user/ozawa/sparkWordcountOutNew2");
java.lang.NullPointerException  


  
at 
org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284)
at 
org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291) 


   
at 
org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480)
at $iwC$$iwC$$iwC$$iwC.(:13) 


  
at $iwC$$iwC$$iwC.(:18)
at $iwC$$iwC.(:20)   


  
at $iwC.(:22)
at (:24) 


  
at .(:28)
at .() 


  
at .(:7)
at .() 


  
at $print()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  


  
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789)   


   
at 
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
at 
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)  


   
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610) 


  
at 
org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:823)
at 
org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:868)


   
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:780)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:625)

[jira] [Updated] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later

2015-01-25 Thread Andrew Or (JIRA) 

 [ 
https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Or updated SPARK-4267:
-
Priority: Blocker  (was: Major)

> Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
> --
>
> Key: SPARK-4267
> URL: https://issues.apache.org/jira/browse/SPARK-4267
> Project: Spark
>  Issue Type: Bug
>Reporter: Tsuyoshi OZAWA
>Priority: Blocker
>
> Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5 
> uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this:
> {code}
>  ./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn 
> -Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0
> {code}
> Then Spark on YARN fails to launch jobs with NPE.
> {code}
> $ bin/spark-shell --master yarn-client
> scala> sc.textFile("hdfs:///user/ozawa/wordcountInput20G").flatMap(line 
> => line.split(" ")).map(word => (word, 1)).persist().reduceByKey((a, b) => a 
> + b, 16).saveAsTextFile("hdfs:///user/ozawa/sparkWordcountOutNew2");
> java.lang.NullPointerException
>   
>   
> 
> at 
> org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284)
> at 
> org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291)   
>   
>   
>  
> at 
> org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480)
> at $iwC$$iwC$$iwC$$iwC.(:13)   
>   
>   
> 
> at $iwC$$iwC$$iwC.(:18)
> at $iwC$$iwC.(:20) 
>   
>   
> 
> at $iwC.(:22)
> at (:24)   
>   
>   
> 
> at .(:28)
> at .()   
>   
>   
> 
> at .(:7)
> at .()   
>   
>   
> 
> at $print()
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   
>   
> 
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   
>   
>   
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789) 
>   
>   
>  
> at 
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
> at 
> org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
>   
>   
>  
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610)   
>   
>   
> 
> at 
> org.apache.spark.repl.SparkILoop.r

[jira] [Updated] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later

2015-01-25 Thread Andrew Or (JIRA) 

 [ 
https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Or updated SPARK-4267:
-
 Component/s: YARN
Target Version/s: 1.3.0

> Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
> --
>
> Key: SPARK-4267
> URL: https://issues.apache.org/jira/browse/SPARK-4267
> Project: Spark
>  Issue Type: Bug
>  Components: YARN
>Reporter: Tsuyoshi OZAWA
>Priority: Blocker
>
> Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5 
> uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this:
> {code}
>  ./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn 
> -Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0
> {code}
> Then Spark on YARN fails to launch jobs with NPE.
> {code}
> $ bin/spark-shell --master yarn-client
> scala> sc.textFile("hdfs:///user/ozawa/wordcountInput20G").flatMap(line 
> => line.split(" ")).map(word => (word, 1)).persist().reduceByKey((a, b) => a 
> + b, 16).saveAsTextFile("hdfs:///user/ozawa/sparkWordcountOutNew2");
> java.lang.NullPointerException
>   
>   
> 
> at 
> org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284)
> at 
> org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291)   
>   
>   
>  
> at 
> org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480)
> at $iwC$$iwC$$iwC$$iwC.(:13)   
>   
>   
> 
> at $iwC$$iwC$$iwC.(:18)
> at $iwC$$iwC.(:20) 
>   
>   
> 
> at $iwC.(:22)
> at (:24)   
>   
>   
> 
> at .(:28)
> at .()   
>   
>   
> 
> at .(:7)
> at .()   
>   
>   
> 
> at $print()
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   
>   
> 
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   
>   
>   
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789) 
>   
>   
>  
> at 
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
> at 
> org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
>   
>   
>  
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610)   
>   
>   
> 
>

[jira] [Updated] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later

2015-02-13 Thread Sean Owen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Owen updated SPARK-4267:
-
Target Version/s:   (was: 1.3.0)
   Fix Version/s: 1.2.2

> Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
> --
>
> Key: SPARK-4267
> URL: https://issues.apache.org/jira/browse/SPARK-4267
> Project: Spark
>  Issue Type: Bug
>  Components: YARN
>Reporter: Tsuyoshi OZAWA
>Assignee: Sean Owen
>Priority: Blocker
> Fix For: 1.3.0, 1.2.2
>
>
> Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5 
> uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this:
> {code}
>  ./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn 
> -Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0
> {code}
> Then Spark on YARN fails to launch jobs with NPE.
> {code}
> $ bin/spark-shell --master yarn-client
> scala> sc.textFile("hdfs:///user/ozawa/wordcountInput20G").flatMap(line 
> => line.split(" ")).map(word => (word, 1)).persist().reduceByKey((a, b) => a 
> + b, 16).saveAsTextFile("hdfs:///user/ozawa/sparkWordcountOutNew2");
> java.lang.NullPointerException
>   
>   
> 
> at 
> org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284)
> at 
> org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291)   
>   
>   
>  
> at 
> org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480)
> at $iwC$$iwC$$iwC$$iwC.(:13)   
>   
>   
> 
> at $iwC$$iwC$$iwC.(:18)
> at $iwC$$iwC.(:20) 
>   
>   
> 
> at $iwC.(:22)
> at (:24)   
>   
>   
> 
> at .(:28)
> at .()   
>   
>   
> 
> at .(:7)
> at .()   
>   
>   
> 
> at $print()
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   
>   
> 
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   
>   
>   
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789) 
>   
>   
>  
> at 
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062)
> at 
> org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615)
>   
>   
>  
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610)   
>   
>