Why spark on yarn applicationmaster cannot get a proper resourcemanager address from yarnconfiguration?

2014-09-03 Thread 남윤민
Hello, I tried to submit a spark job to yarn cluster, there is an error occured with those messages: [root@saturn00 bin]# ./spark-submit --class SparkHiveJoin --master yarn-cluster --num-executors 10 --executor-memory 12g --executor-cores 1 spark.jarSpark assembly has been built with Hive,

Re: Why spark on yarn applicationmaster cannot get a proper resourcemanager address from yarnconfiguration?

2014-09-03 Thread Guodong Wang
Did you follow the exact step in this page https://spark.apache.org/docs/1.0.2/running-on-yarn.html ? Please be sure that HADOOP_CONF_DIR or YARN_CONF_DIR points to the directory which contains the (client side) configuration files for the Hadoop cluster. Guodong On Thu, Sep 4, 2014 at 10:15