ok, we figured it out. It is a bit weird, but for some reason, the YARN_CONF_DIR and HADOOP_CONF_DIR did not propagate out. We do see it in the build classpath, but the remote machines don't seem to get it. So we added: export SPARK_YARN_USER_ENV="CLASSPATH=/hadoop/var/hadoop/conf/"
and it seems to have worked. We also made it work by adding this: export SPARK_YARN_DIST_FILES=$(ls $HADOOP_CONF_DIR* | sed 's#^#file://#g' |tr '\n' ',' ) which distributed the conf dir to all machines. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/YARN-issues-with-resourcemanager-scheduler-address-tp5201p5258.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
