Directory to use for "scratch" space in Spark, including map output files
and RDDs that get stored on disk.
It seems a directory on my local disk. But currently the temp directory is
on hdfs, under the path of df.write.parquet(path)
Regard,
Junfeng Chen
On Wed, Dec 5, 2018 at 6:28 PM Sandip
tryspark.local.dir property.
On Wed, Dec 5, 2018 at 1:42 PM JF Chen wrote:
> I have two spark apps writing data to one directory. I notice they share
> one temp directory, and the spark fist finish writing will clear the temp
> directory and the slower one may throw "No lease on *** File does
I have two spark apps writing data to one directory. I notice they share
one temp directory, and the spark fist finish writing will clear the temp
directory and the slower one may throw "No lease on *** File does not
exist" error
So how to specify the temp directory?
Thanks!
Regard,
Junfeng Chen
