[
https://issues.apache.org/jira/browse/SPARK-44632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-44632:
-----------------------------------
Labels: pull-request-available (was: )
> DiskBlockManager should check and be able to handle stale directories
> ---------------------------------------------------------------------
>
> Key: SPARK-44632
> URL: https://issues.apache.org/jira/browse/SPARK-44632
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.4.1, 3.5.0
> Reporter: Kent Yao
> Priority: Major
> Labels: pull-request-available
>
> The subDir in the memory cache could be stale, for example, after a damaged
> disk repair or replacement. This dir could be accessed subsequently by
> others. Especially, `filename` generated by `RDDBlockId` is unchanged
> between task reties, so it probably attempts to access the same subDir
> repeatedly. Therefore, it is necessary to check if the subDir exists. If it
> is stale and the hardware has been recovered without data and directories, we
> will recreate the subDir to prevent FileNotFoundException during writing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
