[ https://issues.apache.org/jira/browse/HADOOP-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-15583: ------------------------------------ Status: Patch Available (was: Open) > Stabilize S3A Assumed Role support > ---------------------------------- > > Key: HADOOP-15583 > URL: https://issues.apache.org/jira/browse/HADOOP-15583 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.1.0 > Reporter: Steve Loughran > Assignee: Steve Loughran > Priority: Blocker > Attachments: HADOOP-15583-001.patch, HADOOP-15583-002.patch, > HADOOP-15583-003.patch > > > started off just on sharing credentials across S3A and S3Guard, but in the > process it has grown to becoming one of stabilising the assumed role support > so it can be used for more than just testing. > Was: "S3Guard to get AWS Credential chain from S3AFS; credentials closed() on > shutdown" > h3. Issue: lack of auth chain sharing causes ddb and s3 to get out of sync > S3Guard builds its DDB auth chain itself, which stops it having to worry > about being created standalone vs part of an S3AFS, but it means its > authenticators are in a separate chain. > When you are using short-lived assumed roles or other session credentials > updated in the S3A FS authentication chain, you need that same set of > credentials picked up by DDB. Otherwise, at best you are doubling load, at > worse: the DDB connector may not get refreshed credentials. > Proposed: {{DynamoDBClientFactory.createDynamoDBClient()}} to take an > optional ref to aws credentials. If set: don't create a new set. > There's one little complication here: our {{AWSCredentialProviderList}} list > is autocloseable; it's close() will go through all children and close them. > Apparently the AWS S3 client (And hopefully the DDB client) will close this > when they are closed themselves. If DDB has the same set of credentials as > the FS, then there could be trouble if they are closed in one place when the > other still wants to use them. > Solution; have a use count the uses of the credentials list, starting at one: > every close() call decrements, and when this hits zero the cleanup is kicked > off > h3. Issue: {{AssumedRoleCredentialProvider}} connector to STS not picking up > the s3a connection settings, including proxy. > h3. issue: we're not using getPassword() to get user/password for proxy > binding for STS. Fix: use that and pass down the bucket ref for per-bucket > secrets in a JCEKS file. > h3. Issue; hard to debug what's going wrong :) > h3. Issue: docs about KMS permissions for SSE-KMS are wrong, and the > ITestAssumedRole* tests don't request KMS permissions, so fail in a bucket > when the base s3 FS is using SSE-KMS. KMS permissions need to be included in > generated profiles -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org