SparkPi endlessly in yarnAppState: ACCEPTED
I'm using Cloudera 5.1.3, and I'm repeatedly getting the following output after submitting the SparkPi example in yarn cluster mode (http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_running_spark_apps.html) using: spark-submit --class org.apache.spark.examples.SparkPi --deploy-mode cluster --master yarn $SPARK_HOME/examples/lib/spark-examples_2.10-1.0.0-cdh5.1.3.jar 10 Output (repeated): 14/11/07 19:33:05 INFO Client: Application report from ASM: application identifier: application_1415303569855_1100 appId: 1100 clientToAMToken: null appDiagnostics: appMasterHost: N/A appQueue: root.yp appMasterRpcPort: -1 appStartTime: 1415406486231 yarnAppState: ACCEPTED distributedFinalState: UNDEFINED I'll note that spark-submit is working correctly when running with master local on the edge node. Any ideas how to solve this? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkPi-endlessly-in-yarnAppState-ACCEPTED-tp18391.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: SparkPi endlessly in yarnAppState: ACCEPTED
Sounds like no free yarn workers. i.e. try running: hadoop-mapreduce-examples-2.1.0-beta.jar pi 1 1 We have some smoke tests which you might find particularly usefull for yarn clusters as well in https://github.com/apache/bigtop, underneath bigtop-tests/smoke-tests which are generally good to run on any basic hadoop cluster when you first set it up. Often if a Yarn job hangs in accepted state, it is waiting for resources to free up to start the tasks... On Nov 7, 2014, at 7:40 PM, YaoPau jonrgr...@gmail.com wrote: appStartTime
