[
https://issues.apache.org/jira/browse/HADOOP-18544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17640168#comment-17640168
]
Steve Loughran commented on HADOOP-18544:
-----------------------------------------
[~harshit.gupta] assigning to you
this'll need a new s3a option (Constants.java) read to a field in s3afs.
initialize(), then checked in {{maybeCreateFakeParentDirectory()}} to skip the
delete.
then need a test (similar to ITestS3ARenameCost/ITestS3ADeleteCost) which
asserts that no HEAD request is made on rename and delete. those existing
suites will need to set the new option to false to stop all their existing
tests failing; see their parent class's createConfiguration() to see what to do
there.
> S3A: add option to disable probe for dir marker recreation on delete/rename.
> ----------------------------------------------------------------------------
>
> Key: HADOOP-18544
> URL: https://issues.apache.org/jira/browse/HADOOP-18544
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.4
> Reporter: Steve Loughran
> Assignee: Harshit Gupta
> Priority: Major
>
> In applications which do many single-file deletions on the same dir, a lot of
> time is wasted in {{maybeCreateFakeParentDirectory()}}.
> Proposed: add an option to disable the probe, for use by applications which
> are happy for parent dirs to sometimes disappear after a cleanup.
> file by file delete is still woefully inefficient because of the HEAD request
> on every file, but there's no need to amplify the damage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
