[ https://issues.apache.org/jira/browse/HADOOP-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625235#comment-15625235 ]
Steve Loughran commented on HADOOP-13651: ----------------------------------------- # I've actually been talking with [~rajesh.balamohan] about pulling that initial bucket check (HADOOP-13379) . It adds measurable delays to all FS instance creation —and permission errors will show up later on anyway. The tricky bit is having a later 40x failure be uprated to a "there is no such bucket" rather than "you can't access a file". You can save 500+mS by removing an otherwise needless HTTP request; sometimes it can even take longer. I think I would like to cut it, if the failure can be graceful (some tracking of if a request has ever succeeded, on first auth failure, go from simple translation to adding "check bucket exists"/actually falling back to a second check) # if a user has valid read credentials, bucket exist check (currently) fails in init. IF this is delayed, then the first s3 read/write will fail # bucket nonexistent -> 404? 410? -> FNFE # bucket exists but caller not authed -> 401? 403? -> AccessDeniedException # if a user has read but not write credentials, any attempt to do multipart purge will fail; that's now caught & downgraded. # if a user has no credentials, then, if the auth chain has to be set up to allow anonymous access, then they'll try an anonymous auth (not a default option), and they'll get read access to any bucket declared public readable. > S3Guard: S3AFileSystem Integration with MetadataStore > ----------------------------------------------------- > > Key: HADOOP-13651 > URL: https://issues.apache.org/jira/browse/HADOOP-13651 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Reporter: Aaron Fabbri > Assignee: Aaron Fabbri > Attachments: HADOOP-13651-HADOOP-13345.001.patch, > HADOOP-13651-HADOOP-13345.002.patch, HADOOP-13651-HADOOP-13345.003.patch > > > Modify S3AFileSystem et al. to optionally use a MetadataStore for metadata > consistency and caching. > Implementation should have minimal overhead when no MetadataStore is > configured. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org