Raymond: Add addtional: Yes, I build Spark-0.8.1 with -Pnew-yarn, and I followed run-on-yarn.cmd strictly.
Spark web UI shows good for everything. On Tue, Dec 17, 2013 at 12:36 PM, Azuryy Yu <azury...@gmail.com> wrote: > Thanks, Raymond! > My command for Yarn mode: > SPARK_JAR=spark-0.8.1/lib/spark-assembly_2.9.3-0.8.1-incubating-hadoop1.2.1.jar > ./spark-0.8.1/bin/spark-class org.apache.spark.deploy.yarn.Client --jar > spark-0.8.1/spark-examples_2.9.3-0.8.1-incubating.jar --class > org.apache.spark.examples.SparkPi > > > please ingnore hadoop version, it's our customized, which is hadoop-2x > actually. > > but if I don't set spark.driver.*, App Master cannot start, here is the > log: > 13/12/17 11:07:13 INFO yarn.ApplicationMaster: Starting the user JAR in a > separate Thread > 13/12/17 11:07:13 INFO yarn.ApplicationMaster: Waiting for Spark driver to > be reachable. > 13/12/17 11:07:13 WARN yarn.ApplicationMaster: Failed to connect to driver > at null:null, retrying ... > Usage: SparkPi <master> [<slices>] > 13/12/17 11:07:13 WARN yarn.ApplicationMaster: Failed to connect to driver > at null:null, retrying ... > 13/12/17 11:07:13 INFO yarn.ApplicationMaster: AppMaster received a signal. > 13/12/17 11:07:13 WARN yarn.ApplicationMaster: Failed to connect to driver > at null:null, retrying ... > > > After retry 'spark.yarn.applicationMaster.waitTries'(default 10), Job > failed. > > > > > > On Tue, Dec 17, 2013 at 12:07 PM, Liu, Raymond <raymond....@intel.com>wrote: > >> It's what it said on the document. For yarn-standalone mode, it will be >> the host of where spark AM runs, while for yarn-client mode, it will be the >> local host you run the cmd. >> >> And what's cmd you run SparkPi ? I think you actually don't need to set >> sprak.driver.host manually for Yarn mode , SparkContext will handle it for >> you in Automatically and pass it to AM and Executor to use to connect to >> Driver. >> >> Did you follow the guide in docs/running-on-yarn.md ? >> >> >> Best Regards, >> Raymond Liu >> >> From: Azuryy Yu [mailto:azury...@gmail.com] >> Sent: Tuesday, December 17, 2013 11:16 AM >> To: user@spark.incubator.apache.org >> Subject: About spark.driver.host >> >> Hi, >> >> I am using spark-0,8,1, and what's the meaning of spark.driver.host? I >> ran SparkPi failed.(either yarn-standalone or yarn-client) >> >> It was 'Hostname or IP address for the driver to listen on.' in the >> document. but what host the Driver will listen on? the RM on the yarn? if >> yes, I configured spark.driver.host in the spark-env.sh as resource manager >> host and port: >> export SPARK_DAEMON_JAVA_OPTS="-Dspark.driver.host=10.2.8.1 >> -Dspark.driver.port=8032" >> >> but it doesn't work. I find in the log: >> WARN yarn.ApplicationMaster: Failed to connect to driver at null:null, >> retrying ... >> >> Even if I added these two system env variables to the JAVA_OPTS in the >> bin/spark-class, it also doen't work, please help. >> >> Any inputs are appreciated. >> > >
