Kent Yao created SPARK-44632:
--------------------------------
Summary: DiskBlockManager should check and be able to handle stale
directories
Key: SPARK-44632
URL: https://issues.apache.org/jira/browse/SPARK-44632
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.4.1, 3.5.0
Reporter: Kent Yao
The subDir in the memory cache could be stale, for example, after a damaged
disk repair or replacement. This dir could be accessed subsequently by others.
Especially, `filename` generated by `RDDBlockId` is unchanged between task
reties, so it probably attempts to access the same subDir repeatedly.
Therefore, it is necessary to check if the subDir exists. If it is stale and
the hardware has been recovered without data and directories, we will recreate
the subDir to prevent FileNotFoundException during writing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
