Github user mridulm commented on a diff in the pull request:
https://github.com/apache/spark/pull/1609#discussion_r15565447
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala ---
@@ -107,68 +109,296 @@ private[spark] class DiskBlockObjectWriter(
private var fos: FileOutputStream = null
private var ts: TimeTrackingOutputStream = null
private var objOut: SerializationStream = null
+
+ // Did we create this file or was it already present : used in revert to
decide
+ // if we should delete this file or not. Also used to detect if file was
deleted
+ // between creation of BOW and its actual init
+ private val initiallyExists = file.exists() && file.isFile
private val initialPosition = file.length()
private var lastValidPosition = initialPosition
+
private var initialized = false
+ // closed explicitly ?
+ private var closed = false
+ // Attempt to cleanly close ? (could also be closed via revert)
+ // Note, a cleanly closed file could be subsequently reverted
+ private var cleanCloseAttempted = false
+ // Was the file actually opened atleast once.
+ // Note: initialized/streams change state with close/revert.
+ private var wasOpenedOnce = false
private var _timeWriting = 0L
- override def open(): BlockObjectWriter = {
- fos = new FileOutputStream(file, true)
- ts = new TimeTrackingOutputStream(fos)
- channel = fos.getChannel()
+ // Due to some directory creation race issues in spark, it has been
observed that
+ // sometimes file creation happens 'before' the actual directory has
been created
+ // So we attempt to retry atleast once with a mkdirs in case directory
was missing.
--- End diff --
As I mentioned above, we did fix a dcl bug, bug that did not seem
sufficient.
I agree this is a rare condition, and the 'fix' is a hack to workaround the
problem : but pending identifying root cause, this is the best we have
unfortunately.
Any thoughts ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
