You should just be making sure your HADOOP_CONF_DIR env variable is correct and not setting yarn.resourcemanager.address in SparkConf. For Yarn/Hadoop you need to point it to the configuration files for your cluster. Generally that setting goes into yarn-site.xml. If just setting it doesn't work, make sure $HADOOP_CONF_DIR is getting put into your classpath. I would also make sure HADOOP_PREFIX is being set.
Tom On Wednesday, April 2, 2014 10:10 PM, Ron Gonzalez <[email protected]> wrote: Hi, I have a small program but I cannot seem to make it connect to the right properties of the cluster. I have the SPARK_YARN_APP_JAR, SPARK_JAR and SPARK_HOME set properly. If I run this scala file, I am seeing that this is never using the yarn.resourcemanager.address property that I set on the SparkConf instance. Any advice? Thanks, Ron import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ import org.apache.spark.deploy.yarn.Client import java.lang.System import org.apache.spark.SparkConf object SimpleApp { def main(args: Array[String]) { val logFile = "/home/rgonzalez/app/spark-0.9.0-incubating-bin-hadoop2/README.md" val conf = new SparkConf() conf.set("yarn.resourcemanager.address", "localhost:8050") val sc = new SparkContext("yarn-client", "Simple App", conf) val logData = sc.textFile(logFile, 2).cache() val numAs = logData.filter(line => line.contains("a")).count() val numBs = logData.filter(line => line.contains("b")).count() println("Lines with a: %s, Lines with b: %s".format(numAs, numBs)) } }
