Re: Setting Spark TMP Directory in Cluster Mode

2015-09-30 Thread mufy
Any takers? :-) --- Mufeed Usman My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal <http://mufeed.livejournal.com> On Mon, Sep 28, 2015 at 10:19 AM, mufy <mufeed.us...@gmail.com&g

Setting Spark TMP Directory in Cluster Mode

2015-09-25 Thread mufy
Faced with an issue where Spark temp files get filled under /opt/spark-1.2.1/tmp on the local filesystem on the worker nodes. Which parameter/configuration sets that location? The spark-env.sh file has the folders set as, export SPARK_HOME=/opt/spark-1.2.1 export SPARK_WORKER_DIR=$SPARK_HOME/tmp