Yes, you can set the SPARK_LOCAL_DIR in the spark-env.sh or spark.local.dir
in the spark-defaults.conf file, then it would use this location for the
shuffle writes etc.
Thanks
Best Regards
On Wed, Aug 26, 2015 at 6:56 PM, michael.engl...@nomura.com wrote:
Hi,
I am having issues with /tmp
Hi,
I am having issues with /tmp space filling up during Spark jobs because
Spark-on-YARN uses the yarn.nodemanager.local-dirs for shuffle space. I noticed
this message appears when submitting Spark-on-YARN jobs:
WARN SparkConf: In Spark 1.0 and later spark.local.dir will be overridden by
the
