Github user gss2002 commented on the issue:
https://github.com/apache/spark/pull/22867
@vanzin can you review the latest PR. This seems like it works without
issue. I ran this on my home cluster and my larger clusters at work where the
issue was first discovered and it seems to solve
Github user gss2002 commented on the issue:
https://github.com/apache/spark/pull/22867
@vanzin you are right! I appreciate the help with this one. I will cut a
patch in the AM after testing on a large scale cluster job that is taking from
IBM MQ and ETLing data and shipping off to
Github user gss2002 commented on the issue:
https://github.com/apache/spark/pull/22867
@vanzin can you please review latest patch thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user gss2002 commented on the issue:
https://github.com/apache/spark/pull/22867
@vanzin this seems to work.. Not sure what your thoughts are on this
private val tmpDir = broadcastedHadoopConf.value.get("hadoop.tmp.dir",
System.getProperty("
Github user gss2002 commented on the issue:
https://github.com/apache/spark/pull/22867
@vanzin I made the following change and it didn't work. How do you want to
proceed?
---
a/streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala
+
Github user gss2002 commented on the issue:
https://github.com/apache/spark/pull/22867
@vanzin trying this.. I'll advise shortly
private val tmpDir = "file:///" + System.getProperty(
Github user gss2002 commented on the issue:
https://github.com/apache/spark/pull/22867
History from JIRA and error
WriteAheadLogBackedBlockRDD in YARN Cluster Mode Fails due lack of access
to HDFS path
due to it using a similar name was $PWD folder from YARN AM Cluster
Github user gss2002 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22867#discussion_r229130982
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala
---
@@ -136,7 +139,7 @@ class
Github user gss2002 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22867#discussion_r229129992
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala
---
@@ -136,7 +139,7 @@ class
Github user gss2002 commented on the issue:
https://github.com/apache/spark/pull/22867
@lresende any possibilities of getting this change committed or at least
getting it on the radar for key changes for spark to work with YARN Cluster
mode specifically since $PWD creates issues with
GitHub user gss2002 opened a pull request:
https://github.com/apache/spark/pull/22867
[SPARK-25778] WriteAheadLogBackedBlockRDD in YARN Cluster Mode Fails â¦
â¦due lack of access to tmpDir from $PWD to HDFS
WriteAheadLogBackedBlockRDD usage of java.io.tmpdir will fail if
Github user gss2002 commented on the pull request:
https://github.com/apache/spark/pull/2241#issuecomment-56185334
We have been using this fix for a few weeks now against Hive 13. The only
outstanding issue I see and this could be something larger is the fact that
Spark Thrift
12 matches
Mail list logo
