[
https://issues.apache.org/jira/browse/HADOOP-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17191816#comment-17191816
]
Steve Loughran commented on HADOOP-13230:
-----------------------------------------
First followup JIRA to this. HADOOP-17244
> S3A to optionally retain directory markers
> ------------------------------------------
>
> Key: HADOOP-13230
> URL: https://issues.apache.org/jira/browse/HADOOP-13230
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0
> Reporter: Aaron Fabbri
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.1
>
> Attachments: 2020-02-Fixing the S3A directory marker problem.pdf
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Users of s3a may not realize that, in some cases, it does not interoperate
> well with other s3 tools, such as the AWS CLI. (See HIVE-13778, IMPALA-3558).
> Specifically, if a user:
> - Creates an empty directory with hadoop fs -mkdir s3a://bucket/path
> - Copies data into that directory via another tool, i.e. aws cli.
> - Tries to access the data in that directory with any Hadoop software.
> Then the last step fails because the fake empty directory blob that s3a wrote
> in the first step, causes s3a (listStatus() etc.) to continue to treat that
> directory as empty, even though the second step was supposed to populate the
> directory with data.
> I wanted to document this fact for users. We may mark this as not-fix, "by
> design".. May also be interesting to brainstorm solutions and/or a config
> option to change the behavior if folks care.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
