[jira] [Updated] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
[ https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated SPARK-4267: -- Description: Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5 uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this: {code} ./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn -Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0 {code} Then Spark on YARN cannot fail to run with NPE. {code} $ bin/spark-shell --master yarn-client scala> sc.textFile("hdfs:///user/ozawa/wordcountInput20G").flatMap(line => line.split(" ")).map(word => (word, 1)).persist().reduceByKey((a, b) => a + b, 16).saveAsTextFile("hdfs:///user/ozawa/sparkWordcountOutNew2"); java.lang.NullPointerException at org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284) at org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291) at org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480) at $iwC$$iwC$$iwC$$iwC.(:13) at $iwC$$iwC$$iwC.(:18) at $iwC$$iwC.(:20) at $iwC.(:22) at (:24) at .(:28) at .() at .(:7) at .() at $print() at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789) at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062) at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610) at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:823) at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:868) at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:780) at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:625)
[jira] [Updated] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
[ https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated SPARK-4267: -- Description: Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5 uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this: {code} ./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn -Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0 {code} Then Spark on YARN fails to launch jobs with NPE. {code} $ bin/spark-shell --master yarn-client scala> sc.textFile("hdfs:///user/ozawa/wordcountInput20G").flatMap(line => line.split(" ")).map(word => (word, 1)).persist().reduceByKey((a, b) => a + b, 16).saveAsTextFile("hdfs:///user/ozawa/sparkWordcountOutNew2"); java.lang.NullPointerException at org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284) at org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291) at org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480) at $iwC$$iwC$$iwC$$iwC.(:13) at $iwC$$iwC$$iwC.(:18) at $iwC$$iwC.(:20) at $iwC.(:22) at (:24) at .(:28) at .() at .(:7) at .() at $print() at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789) at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062) at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610) at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:823) at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:868) at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:780) at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:625)
[jira] [Updated] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
[ https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-4267: - Priority: Blocker (was: Major) > Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later > -- > > Key: SPARK-4267 > URL: https://issues.apache.org/jira/browse/SPARK-4267 > Project: Spark > Issue Type: Bug >Reporter: Tsuyoshi OZAWA >Priority: Blocker > > Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5 > uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this: > {code} > ./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn > -Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0 > {code} > Then Spark on YARN fails to launch jobs with NPE. > {code} > $ bin/spark-shell --master yarn-client > scala> sc.textFile("hdfs:///user/ozawa/wordcountInput20G").flatMap(line > => line.split(" ")).map(word => (word, 1)).persist().reduceByKey((a, b) => a > + b, 16).saveAsTextFile("hdfs:///user/ozawa/sparkWordcountOutNew2"); > java.lang.NullPointerException > > > > at > org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284) > at > org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291) > > > > at > org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480) > at $iwC$$iwC$$iwC$$iwC.(:13) > > > > at $iwC$$iwC$$iwC.(:18) > at $iwC$$iwC.(:20) > > > > at $iwC.(:22) > at (:24) > > > > at .(:28) > at .() > > > > at .(:7) > at .() > > > > at $print() > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789) > > > > at > org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062) > at > org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615) > > > > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646) > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610) > > > > at > org.apache.spark.repl.SparkILoop.r
[jira] [Updated] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
[ https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-4267: - Component/s: YARN Target Version/s: 1.3.0 > Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later > -- > > Key: SPARK-4267 > URL: https://issues.apache.org/jira/browse/SPARK-4267 > Project: Spark > Issue Type: Bug > Components: YARN >Reporter: Tsuyoshi OZAWA >Priority: Blocker > > Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5 > uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this: > {code} > ./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn > -Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0 > {code} > Then Spark on YARN fails to launch jobs with NPE. > {code} > $ bin/spark-shell --master yarn-client > scala> sc.textFile("hdfs:///user/ozawa/wordcountInput20G").flatMap(line > => line.split(" ")).map(word => (word, 1)).persist().reduceByKey((a, b) => a > + b, 16).saveAsTextFile("hdfs:///user/ozawa/sparkWordcountOutNew2"); > java.lang.NullPointerException > > > > at > org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284) > at > org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291) > > > > at > org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480) > at $iwC$$iwC$$iwC$$iwC.(:13) > > > > at $iwC$$iwC$$iwC.(:18) > at $iwC$$iwC.(:20) > > > > at $iwC.(:22) > at (:24) > > > > at .(:28) > at .() > > > > at .(:7) > at .() > > > > at $print() > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789) > > > > at > org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062) > at > org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615) > > > > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646) > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610) > > > >
[jira] [Updated] (SPARK-4267) Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later
[ https://issues.apache.org/jira/browse/SPARK-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-4267: - Target Version/s: (was: 1.3.0) Fix Version/s: 1.2.2 > Failing to launch jobs on Spark on YARN with Hadoop 2.5.0 or later > -- > > Key: SPARK-4267 > URL: https://issues.apache.org/jira/browse/SPARK-4267 > Project: Spark > Issue Type: Bug > Components: YARN >Reporter: Tsuyoshi OZAWA >Assignee: Sean Owen >Priority: Blocker > Fix For: 1.3.0, 1.2.2 > > > Currently we're trying Spark on YARN included in Hadoop 2.5.1. Hadoop 2.5 > uses protobuf 2.5.0 so I compiled with protobuf 2.5.1 like this: > {code} > ./make-distribution.sh --name spark-1.1.1 --tgz -Pyarn > -Dhadoop.version=2.5.1 -Dprotobuf.version=2.5.0 > {code} > Then Spark on YARN fails to launch jobs with NPE. > {code} > $ bin/spark-shell --master yarn-client > scala> sc.textFile("hdfs:///user/ozawa/wordcountInput20G").flatMap(line > => line.split(" ")).map(word => (word, 1)).persist().reduceByKey((a, b) => a > + b, 16).saveAsTextFile("hdfs:///user/ozawa/sparkWordcountOutNew2"); > java.lang.NullPointerException > > > > at > org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1284) > at > org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:1291) > > > > at > org.apache.spark.SparkContext.textFile$default$2(SparkContext.scala:480) > at $iwC$$iwC$$iwC$$iwC.(:13) > > > > at $iwC$$iwC$$iwC.(:18) > at $iwC$$iwC.(:20) > > > > at $iwC.(:22) > at (:24) > > > > at .(:28) > at .() > > > > at .(:7) > at .() > > > > at $print() > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:789) > > > > at > org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1062) > at > org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:615) > > > > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:646) > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:610) > >