Re: Setting Spark TMP Directory in Cluster Mode

Any takers? :-) --- Mufeed Usman My LinkedIn | My Social Cause | My Blogs : LiveJournal On Mon, Sep 28, 2015 at 10:19 AM, mufy wrote: > Hello Akhil,

Setting Spark TMP Directory in Cluster Mode

Faced with an issue where Spark temp files get filled under /opt/spark-1.2.1/tmp on the local filesystem on the worker nodes. Which parameter/configuration sets that location? The spark-env.sh file has the folders set as, export SPARK_HOME=/opt/spark-1.2.1 export SPARK_WORKER_DIR=$SPARK_HOME/tmp

Re: Setting Spark TMP Directory in Cluster Mode

Try with spark.local.dir in the spark-defaults.conf or SPARK_LOCAL_DIR in the spark-env.sh file. Thanks Best Regards On Fri, Sep 25, 2015 at 2:14 PM, mufy wrote: > Faced with an issue where Spark temp files get filled under > /opt/spark-1.2.1/tmp on the local filesystem