[
https://issues.apache.org/jira/browse/HADOOP-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16976082#comment-16976082
]
Steve Loughran commented on HADOOP-16711:
-----------------------------------------
Please justify.
* If you believe it is because you don't need to check with things being there
until you actually do some form of file IO, that is quite an interesting
change. it means errors about non-existent buckets will only surface when you
open a nonzero byte file to read, or try to write something. I'm curious
whether we would sensibly translate whatever error comes back into a one about
the bucket rather than the file.
* If it is because S3Guard + DDB itself checks for the bucket location -that
only happens if you don't specify region. But it will be nice to consider how
we could combine the two checks. But remember you can have a different
implementations of S3Guard; the local one does not talk AWS at all.
* Be aware - the AWS S3 SDK apparently does some calls too, so that it can
determine which endpoints to use for v4 signing. I believe that is a recent
change in the SDKs -as Sidd Seth about it.
If you're worried about start-up latency you are better off having an
application create the S3 client asynchronously. That would hide all the
overhead of DNS lookups, DDB init, S3 probes in both ours and the AWS libraries.
See ORC-570
+ please tag with version, components etc.
> With S3Guard + authmode, consider skipping "verifyBuckets" check in S3A fs
> init()
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-16711
> URL: https://issues.apache.org/jira/browse/HADOOP-16711
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Rajesh Balamohan
> Priority: Minor
> Labels: performance
> Attachments: HADOOP-16711.prelim.1.patch
>
>
> When authoritative mode is enabled with s3guard, it would be good to skip
> verifyBuckets call during S3A filesystem init(). This would save call to S3
> during init method.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
