GitHub user gss2002 opened a pull request:
https://github.com/apache/spark/pull/22867
[SPARK-25778] WriteAheadLogBackedBlockRDD in YARN Cluster Mode Fails â¦
â¦due lack of access to tmpDir from $PWD to HDFS
WriteAheadLogBackedBlockRDD usage of java.io.tmpdir will fail if $PWD
resolves to a folder in HDFS and the Spark YARN Cluster job does not have the
correct access to this folder in regards to the dummy folder. So this patch
provides an option to set spark.streaming.receiver.blockStore.tmpdir to
override java.io.tmpdir which sets $PWD from YARN Cluster mode.
## What changes were proposed in this pull request?
This change provides an option to override the java.io.tmpdir option so
that when $PWD is resolved in YARN Cluster mode Spark does not attempt to use
this folder and instead use the folder provided with the following option:
spark.streaming.receiver.blockStore.tmpdir
## How was this patch tested?
Patch was manually tested on a Spark Streaming Job with Write Ahead logs in
Cluster mode.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gss2002/spark SPARK-25778
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22867.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22867
----
commit 4970827038fb18f74c1b7975d7bfc00609dc9405
Author: gss2002 <greg@...>
Date: 2018-10-28T04:22:31Z
[SPARK-25778] WriteAheadLogBackedBlockRDD in YARN Cluster Mode Fails due
lack of access to tmpDir from $PWD to HDFS
WriteAheadLogBackedBlockRDD usage of java.io.tmpdir will fail if $PWD
resolves to a folder in HDFS and the Spark YARN Cluster job does not have the
correct access to this folder in regards to the dummy folder. So this patch
provides an option to set spark.streaming.receiver.blockStore.tmpdir to
override java.io.tmpdir which sets $PWD from YARN Cluster mode.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
