[
https://issues.apache.org/jira/browse/HADOOP-18636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17818777#comment-17818777
]
ASF GitHub Bot commented on HADOOP-18636:
-----------------------------------------
lowezheng commented on PR #5412:
URL: https://github.com/apache/hadoop/pull/5412#issuecomment-1954058932
${hadoop.tmp.dir} is set to "/tmp/hadoop-root" by default. This path can be
deleted by the operating system or manually. This will cause the above
problems. I think the same logic needs to be added to the screenshot below in
the code. in version 3.3.6
<img width="1041" alt="image"
src="https://github.com/apache/hadoop/assets/1687284/6c687c7f-77b7-4f14-bcda-baa62cdc88b3";>
> LocalDirAllocator cannot recover from directory tree deletion during the life
> of a filesystem client
> ----------------------------------------------------------------------------------------------------
>
> Key: HADOOP-18636
> URL: https://issues.apache.org/jira/browse/HADOOP-18636
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/azure, fs/s3
> Affects Versions: 3.3.4
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.6
>
>
> The s3a and abfs clients use LocalDirAllocator for allocating files in local
> (temporary) storage for buffering blocks to write, and, for the s3a staging
> committer, files being staged.
> When initialized (or when the configuration key value is updated)
> LocalDirAllocator enumerates all directories in the list and calls
> {{mkdirs()}} to create them.
> when you ask actually for a file, it will look for the parent dir, and will
> again call {{mkdirs()}}.
> But before it does that, it looks to see if the dir has any space...if not it
> is excluded from the list of directories with room for data.
> And guess what: directories which don't exist report as having no space. So
> they get excluded -the recreation code doesn't get a chance to run.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
