As a debugging step, does it work if you use a single resource manager with the key "yarn.resourcemanager.address" instead of using two named resource managers? I wonder if somehow the YARN client can't detect this multi-master set-up.
On Sun, Jun 1, 2014 at 12:49 PM, Xu (Simon) Chen <xche...@gmail.com> wrote: > Note that everything works fine in spark 0.9, which is packaged in CDH5: I > can launch a spark-shell and interact with workers spawned on my yarn > cluster. > > So in my /opt/hadoop/conf/yarn-site.xml, I have: > ... > <property> > <name>yarn.resourcemanager.address.rm1</name> > <value>controller-1.mycomp.com:23140</value> > </property> > ... > <property> > <name>yarn.resourcemanager.address.rm2</name> > <value>controller-2.mycomp.com:23140</value> > </property> > ... > > And the other usual stuff. > > So spark 1.0 is launched like this: > Spark Command: java -cp > ::/home/chenxu/spark-1.0.0-bin-hadoop2/conf:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/spark-assembly-1.0.0-hadoop2.2.0.jar:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/datanucleus-core-3.2.2.jar:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/datanucleus-api-jdo-3.2.1.jar:/home/chenxu/spark-1.0.0-bin-hadoop2/lib/datanucleus-rdbms-3.2.1.jar:/opt/hadoop/conf > -XX:MaxPermSize=128m -Djava.library.path= -Xms512m -Xmx512m > org.apache.spark.deploy.SparkSubmit spark-shell --master yarn-client --class > org.apache.spark.repl.Main > > I do see "/opt/hadoop/conf" included, but not sure it's the right place. > > Thanks.. > -Simon > > > > On Sun, Jun 1, 2014 at 1:57 PM, Patrick Wendell <pwend...@gmail.com> wrote: >> >> I would agree with your guess, it looks like the yarn library isn't >> correctly finding your yarn-site.xml file. If you look in >> yarn-site.xml do you definitely the resource manager >> address/addresses? >> >> Also, you can try running this command with >> SPARK_PRINT_LAUNCH_COMMAND=1 to make sure the classpath is being >> set-up correctly. >> >> - Patrick >> >> On Sat, May 31, 2014 at 5:51 PM, Xu (Simon) Chen <xche...@gmail.com> >> wrote: >> > Hi all, >> > >> > I tried a couple ways, but couldn't get it to work.. >> > >> > The following seems to be what the online document >> > (http://spark.apache.org/docs/latest/running-on-yarn.html) is >> > suggesting: >> > >> > SPARK_JAR=hdfs://test/user/spark/share/lib/spark-assembly-1.0.0-hadoop2.2.0.jar >> > YARN_CONF_DIR=/opt/hadoop/conf ./spark-shell --master yarn-client >> > >> > Help info of spark-shell seems to be suggesting "--master yarn >> > --deploy-mode >> > cluster". >> > >> > But either way, I am seeing the following messages: >> > 14/06/01 00:33:20 INFO client.RMProxy: Connecting to ResourceManager at >> > /0.0.0.0:8032 >> > 14/06/01 00:33:21 INFO ipc.Client: Retrying connect to server: >> > 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is >> > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) >> > 14/06/01 00:33:22 INFO ipc.Client: Retrying connect to server: >> > 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is >> > RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) >> > >> > My guess is that spark-shell is trying to talk to resource manager to >> > setup >> > spark master/worker nodes - I am not sure where 0.0.0.0:8032 came from >> > though. I am running CDH5 with two resource managers in HA mode. Their >> > IP/port should be in /opt/hadoop/conf/yarn-site.xml. I tried both >> > HADOOP_CONF_DIR and YARN_CONF_DIR, but that info isn't picked up. >> > >> > Any ideas? Thanks. >> > -Simon > >
