steveloughran commented on issue #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client. URL: https://github.com/apache/hadoop/pull/1826#issuecomment-582574788 BTW, latest patch adds an experimental "optimize directory markers" switch which tells the S3a client to be less strict about creating and deleting directory markers. I'm explicitly vague about what that means, but currently in file creation it only ever looks one level up to delete markers. There's trouble there if applications don't call mkdir() on a path before creating the file, but otherwise it avoids the tombstones and scale problems on deep trees
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
