[
https://issues.apache.org/jira/browse/HADOOP-17244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17191814#comment-17191814
]
Steve Loughran commented on HADOOP-17244:
-----------------------------------------
Working hypothesis
The presence of a marker with child entries in s3guard *which don't have S3
entries of their own* causes the marker to be deleted in the final batch of
paths deleted (maybe also at an interim point), which then places a tombstone
at that point in the table.
the subsequent scan to delete all s3guard entries hits the tombstone, so
doesn't bother looking underneath.
When you create the new dir back again, then the orphan entry is reinstated.so
it reappears as a directory.
Now, why only in auth mode? The bucket is set up with auth mode retention
(albeit for only a different subdir of the store). Once the -Dauth test run
puts the bucket into auth mode, its retained everywhere, and this test setup
finds it.
{code}
<property>
<name>fs.s3a.bucket.stevel-london.directory.marker.retention</name>
<value>authoritative</value>
</property>
<property>
<name>fs.s3a.bucket.stevel-london.authoritative.path</name>
<value>/tables</value>
</property>
{code}
Now #2: so why didn't -Dkeep find it? No idea whatsoever. Maybe it's also that
the auth flag changes listing too
> ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory
> test failure on -Dauth
> --------------------------------------------------------------------------------------------------
>
> Key: HADOOP-17244
> URL: https://issues.apache.org/jira/browse/HADOOP-17244
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Blocker
>
> Test failure:
> {{ITestS3AFileContextMainOperations#testRenameDirectoryAsNonExistentDirectory}}
> This is repeatable on -Dauth runs (we haven't been running them, have we?)
> Either its from the recent dir marker changes (initial hypothesis) or its
> been lurking a while and not been picked up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
