ok, we figured it out. It is a bit weird, but for some reason, the
YARN_CONF_DIR and HADOOP_CONF_DIR did not propagate out. We do see it in
the build classpath, but the remote machines don't seem to get it. So we
added:
export SPARK_YARN_USER_ENV=CLASSPATH=/hadoop/var/hadoop/conf/
and it seems
Hi,
I'm trying to connect to a YARN cluster by running these commands:
export HADOOP_CONF_DIR=/hadoop/var/hadoop/conf/
export YARN_CONF_DIR=$HADOOP_CONF_DIR
export SPARK_YARN_MODE=true
export
SPARK_JAR=./assembly/target/scala-2.10/spark-assembly_2.10-0.9.1-hadoop2.2.0.jar
export