[ https://issues.apache.org/jira/browse/HADOOP-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HADOOP-13230: ------------------------------------ Labels: pull-request-available (was: ) > S3A to optionally retain directory markers > ------------------------------------------ > > Key: HADOOP-13230 > URL: https://issues.apache.org/jira/browse/HADOOP-13230 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 2.9.0 > Reporter: Aaron Fabbri > Assignee: Steve Loughran > Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Attachments: 2020-02-Fixing the S3A directory marker problem.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Users of s3a may not realize that, in some cases, it does not interoperate > well with other s3 tools, such as the AWS CLI. (See HIVE-13778, IMPALA-3558). > Specifically, if a user: > - Creates an empty directory with hadoop fs -mkdir s3a://bucket/path > - Copies data into that directory via another tool, i.e. aws cli. > - Tries to access the data in that directory with any Hadoop software. > Then the last step fails because the fake empty directory blob that s3a wrote > in the first step, causes s3a (listStatus() etc.) to continue to treat that > directory as empty, even though the second step was supposed to populate the > directory with data. > I wanted to document this fact for users. We may mark this as not-fix, "by > design".. May also be interesting to brainstorm solutions and/or a config > option to change the behavior if folks care. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org