Re: hadoop_conf_dir when running spark on yarn
Hi, On Mon, Nov 3, 2014 at 1:29 PM, Amey Chaugule ambr...@gmail.com wrote: I thought that only applied when you're trying to run a job using spark-submit or in the shell... And how are you starting your Yarn job, if not via spark-submit? Tobias
Re: hadoop_conf_dir when running spark on yarn
You can set HADOOP_CONF_DIR inside the spark-env.sh file Thanks Best Regards On Sat, Nov 1, 2014 at 4:14 AM, ameyc ambr...@gmail.com wrote: How do i setup hadoop_conf_dir correctly when I'm running my spark job on Yarn? My Yarn environment has the correct hadoop_conf_dir settings by the configuration that I pull from sc.hadoopConfiguration() is incorrect. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/hadoop-conf-dir-when-running-spark-on-yarn-tp17872.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: hadoop_conf_dir when running spark on yarn
I thought that only applied when you're trying to run a job using spark-submit or in the shell... On Sun, Nov 2, 2014 at 8:47 AM, Akhil Das ak...@sigmoidanalytics.com wrote: You can set HADOOP_CONF_DIR inside the spark-env.sh file Thanks Best Regards On Sat, Nov 1, 2014 at 4:14 AM, ameyc ambr...@gmail.com wrote: How do i setup hadoop_conf_dir correctly when I'm running my spark job on Yarn? My Yarn environment has the correct hadoop_conf_dir settings by the configuration that I pull from sc.hadoopConfiguration() is incorrect. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/hadoop-conf-dir-when-running-spark-on-yarn-tp17872.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- Amey S. Chaugule B.S Computer Engineering and Mathematics '11, University of Illinois at Urbana-Champaign
hadoop_conf_dir when running spark on yarn
How do i setup hadoop_conf_dir correctly when I'm running my spark job on Yarn? My Yarn environment has the correct hadoop_conf_dir settings by the configuration that I pull from sc.hadoopConfiguration() is incorrect. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/hadoop-conf-dir-when-running-spark-on-yarn-tp17872.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
