Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22867#discussion_r229129104
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/rdd/WriteAheadLogBackedBlockRDD.scala
---
@@ -136,7 +139,7 @@ class WriteAheadLogBackedBlockRDD[T: ClassTag](
// this dummy directory should not already exist otherwise the WAL
will try to recover
// past events from the directory and throw errors.
val nonExistentDirectory = new File(
- System.getProperty("java.io.tmpdir"),
UUID.randomUUID().toString).getAbsolutePath
+ tmpDir, UUID.randomUUID().toString).getAbsolutePath
--- End diff --
I don't think a new configuration is the right thing here.
Seems like the problem here is that this should really be pointing at the
local disk, and not `fs.defaultFS`; in which case, instead of
`getAbsolutePath`, it should be `.toURI().toString()`. Does that work?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
