Sam Kramer created HADOOP-18278:
-----------------------------------
Summary: Do not perform a LIST call when creating a file
Key: HADOOP-18278
URL: https://issues.apache.org/jira/browse/HADOOP-18278
Project: Hadoop Common
Issue Type: Improvement
Reporter: Sam Kramer
Hello,
We've noticed that when creating a file, which does not exist in S3, we see an
extra LIST call gets issued to see if it's a directory (i.e. if key = "bar", it
will issue an object list request for "bar/").
Is this really necessary, shouldn't a HEAD request be sufficient to determine
if it actually exists or not? As we're creating 1000s of files, this is quite
expensive, as we're effectively doubling our costs for file creation. Curious
if others have experienced similar or identical issues, or if there are any
workarounds.
[https://github.com/apache/hadoop/blob/516a2a8e440378c868ddb02cb3ad14d0d879037f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3359-L3369]
Thanks,
Sam
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
