[ https://issues.apache.org/jira/browse/HADOOP-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-15583: ------------------------------------ Status: Open (was: Patch Available) > S3Guard to get AWS Credential chain from S3AFS; credentials closed() on > shutdown > -------------------------------------------------------------------------------- > > Key: HADOOP-15583 > URL: https://issues.apache.org/jira/browse/HADOOP-15583 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.1.0 > Reporter: Steve Loughran > Assignee: Steve Loughran > Priority: Major > Attachments: HADOOP-15583-001.patch, HADOOP-15583-002.patch > > > S3Guard builds its DDB auth chain itself, which stops it having to worry > about being created standalone vs part of an S3AFS, but it means its > authenticators are in a separate chain. > When you are using short-lived assumed roles or other session credentials > updated in the S3A FS authentication chain, you need that same set of > credentials picked up by DDB. Otherwise, at best you are doubling load, at > worse: the DDB connector may not get refreshed credentials. > Proposed: {{DynamoDBClientFactory.createDynamoDBClient()}} to take an > optional ref to aws credentials. If set: don't create a new set. > There's one little complication here: our {{AWSCredentialProviderList}} list > is autocloseable; it's close() will go through all children and close them. > Apparently the AWS S3 client (And hopefully the DDB client) will close this > when they are closed themselves. If DDB has the same set of credentials as > the FS, then there could be trouble if they are closed in one place when the > other still wants to use them. > Solution; have a use count the uses of the credentials list, starting at one: > every close() call decrements, and when this hits zero the cleanup is kicked > off -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org