Trying to run Spark on Yarn

I'm running into an authentication issue when running against YARN. I am using my own method to create the JAR assembly file and most likely I am missing something. This method used to work, but I recently ran into this problem. Here is the error from the YARN server: 14/05/23 19:03:02 INFO

Re: YARN issues with resourcemanager.scheduler.address

ok, we figured it out. It is a bit weird, but for some reason, the YARN_CONF_DIR and HADOOP_CONF_DIR did not propagate out. We do see it in the build classpath, but the remote machines don't seem to get it. So we added: export SPARK_YARN_USER_ENV=CLASSPATH=/hadoop/var/hadoop/conf/ and it seems

YARN issues with resourcemanager.scheduler.address

Hi, I'm trying to connect to a YARN cluster by running these commands: export HADOOP_CONF_DIR=/hadoop/var/hadoop/conf/ export YARN_CONF_DIR=$HADOOP_CONF_DIR export SPARK_YARN_MODE=true export SPARK_JAR=./assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar export