[
https://issues.apache.org/jira/browse/HADOOP-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095080#comment-18095080
]
Steve Loughran commented on HADOOP-19937:
-----------------------------------------
Marker retention was cut in HADOOP-19278 and won't go back
- deletion destroys performance on versioned buckets, and you won't even notice
what the problem is
- twin codepaths makes for uglier code and double test time.
we could probably expand to doing the HEAD as well as the LIST; that seems to
make sense. Anything else though, sorry, wontfix
> When a folder is empty or renamed, fs.s3a.directory.marker.retention=delete
> should delete the parent markers
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-19937
> URL: https://issues.apache.org/jira/browse/HADOOP-19937
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.5.0, 3.4.1
> Reporter: Jayesh Lalwani
> Priority: Minor
>
> Right now, when a folder is deleted or renamed, S3A creates the parent
> markers for the empty folders. This makes sense when
> fs.s3a.directory.marker.retention is set to keep.
>
> However, when fs.s3a.directory.marker.retention is set to delete, it should
> skip creation of parent markers to maintain backward compatibility.
>
> {*}Code references (branch hadoop-3.4.2):{*}{*}{*}
>
> - S3AFileSystem.deleteWithoutCloseCheck() (line 4022) unconditionally calls
> maybeCreateFakeParentDirectory(path) after delete
> - OperationCallbacksImpl.finishRename() (line 3091) unconditionally calls
> maybeCreateFakeParentDirectory(sourceRenamed) after rename
> - Neither call checks keepDirectoryMarkers() before creating the marker
>
> {*}Impact:{*}{*}{*}
>
> Customers migrating from EMRFS to S3A (e.g., EMR 7.8 → 7.10) see orphan
> zero-byte directory markers accumulating after FileSystem.rename()
> operations. These markers cause applications (e.g., Spark Structured
> Streaming) to issue nnecessary S3 LIST calls on empty paths, increasing S3
> costs. Setting fs.s3a.directory.marker.retention=delete does not prevent this
> behavior.
>
> {*}Proposed fix:{*}{*}{*}
>
> Guard the maybeCreateFakeParentDirectory() calls with the marker retention
> policy:
>
> {{// In finishRename():}}
> {{if (keepDirectoryMarkers(sourceRenamed.getParent())) {}}
> {{ maybeCreateFakeParentDirectory(sourceRenamed);}}
> {{ }}}
>
> {{// In deleteWithoutCloseCheck():}}
> {{if (keepDirectoryMarkers(path.getParent())) {}}
> {{ maybeCreateFakeParentDirectory(path);}}
> {{ }}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
