Hi, I've stuck using either yarn-client or standalone-client mode. Either will stuck when I submit jobs, the last messages it printed were:
... 14/06/17 02:37:17 INFO spark.SparkContext: Added JAR file:/x/home/jianshuang/tmp/lib/commons-vfs2.jar at http://10.196.195.25:56377/jars/commons-vfs2.jar with timestamp 1402997837065 14/06/17 02:37:17 INFO spark.SparkContext: Added JAR file:/x/home/jianshuang/tmp/rtgraph.jar at http://10.196.195.25:56377/jars/rtgraph.jar with timestamp 1402997837065 14/06/17 02:37:17 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler 14/06/17 02:37:17 INFO yarn.ApplicationMaster$$anon$1: Adding shutdown hook for context org.apache.spark.SparkContext@6655cf60 I can use yarn-cluster to run my app but it's not very convenient to monitor the progress. Standalone-cluster mode doesn't work, it reports file not found error: Driver successfully submitted as driver-20140617023956-0003 ... waiting before polling master for driver state ... polling master for driver state State of driver-20140617023956-0003 is ERROR Exception from cluster was: java.io.FileNotFoundException: File file:/x/home/jianshuang/tmp/rtgraph.jar does not exist I'm using Spark 1.0.0 and my submit command looks like this: ~/spark/spark-1.0.0-hadoop2.4.0/bin/spark-submit --name 'rtgraph' --class com.paypal.rtgraph.demo.MapReduceWriter --master spark:// lvshdc5en0015.lvs.paypal.com:7077 --jars `find lib -type f | tr '\n' ','` --executor-memory 20G --total-executor-cores 96 --deploy-mode cluster rtgraph.jar List of jars I put in --jars option are: accumulo-core.jar accumulo-fate.jar accumulo-minicluster.jar accumulo-trace.jar accumulo-tracer.jar chill_2.10-0.3.6.jar commons-math.jar commons-vfs2.jar config-1.2.1.jar gson.jar guava.jar joda-convert-1.2.jar joda-time-2.3.jar kryo-2.21.jar libthrift.jar quasiquotes_2.10-2.0.0-M8.jar scala-async_2.10-0.9.1.jar scala-library-2.10.4.jar scala-reflect-2.10.4.jar Anyone has hint what went wrong? Really confused. Cheers, -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/
