Hi Expert, Hadoop version: 2.4 Spark version: 1.3.1
I am running the SparkPi example application. bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --executor-memory 2G lib/spark-examples-1.3.1-hadoop2.4.0.jar 20000 The same command sometimes gets WARN ReliableDeliverySupervisor, sometimes does not. Some runs are successful even with the WARN bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --executor-memory 2G lib/spark-examples-1.3.1-hadoop2.4.0.jar 10000 15/07/02 04:38:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Pi is roughly 3.141633956 bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --executor-memory 2G lib/spark-examples-1.3.1-hadoop2.4.0.jar 20000 15/07/02 05:17:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/07/02 05:17:53 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkYarnAM@hostname:32544] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 15/07/02 05:18:01 ERROR YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED! Exception in thread "main" java.lang.NullPointerException at org.apache.spark.SparkContext.<init>(SparkContext.scala:544) at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:28) at org.apache.spark.examples.SparkPi.main(SparkPi.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --executor-memory 2G lib/spark-examples-1.3.1-hadoop2.4.0.jar 10000 15/07/02 05:23:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/07/02 05:24:09 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkYarnAM@hostname:15959] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. Pi is roughly 3.141625776 Also, the spark ui only available when I set --master to local. What could have caused those issues ? Thanks, Xiaohe