[ https://issues.apache.org/jira/browse/HADOOP-16806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17582973#comment-17582973 ]
ASF GitHub Bot commented on HADOOP-16806: ----------------------------------------- jmahonin commented on PR #4753: URL: https://github.com/apache/hadoop/pull/4753#issuecomment-1222353654 Tried with and without an `fs.s3a.assumed.role.arn` setting. I'm not sure if I'm expected to set up this dataset manually, or if there's some magic behind the scenes that should make it available? ``` [ERROR] testJobSubmissionCollectsTokens[0](org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob) Time elapsed: 10.84 s <<< ERROR! org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on s3a://landsat-pds/scene_list.gz: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 16AXE4CXSVZBN8BH; S3 Extended Request ID: 3F4W4GXGPmUqh9fgojrQoGZYInEGXbKez4G3LAzmbSvL/1nyyyAq0kfXrr5LszB5B6wg7Inr6SM=; Proxy: null), S3 Extended Request ID: 3F4W4GXGPmUqh9fgojrQoGZYInEGXbKez4G3LAzmbSvL/1nyyyAq0kfXrr5LszB5B6wg7Inr6SM=:400 Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 16AXE4CXSVZBN8BH; S3 Extended Request ID: 3F4W4GXGPmUqh9fgojrQoGZYInEGXbKez4G3LAzmbSvL/1nyyyAq0kfXrr5LszB5B6wg7Inr6SM=; Proxy: null) at org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.testJobSubmissionCollectsTokens(ITestDelegatedMRJob.java:281) Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 16AXE4CXSVZBN8BH; S3 Extended Request ID: 3F4W4GXGPmUqh9fgojrQoGZYInEGXbKez4G3LAzmbSvL/1nyyyAq0kfXrr5LszB5B6wg7Inr6SM=; Proxy: null) at org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.testJobSubmissionCollectsTokens(ITestDelegatedMRJob.java:281) [ERROR] testJobSubmissionCollectsTokens[1](org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob) Time elapsed: 7.758 s <<< ERROR! org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on s3a://landsat-pds/scene_list.gz: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: W01GKMAXC95KHVPR; S3 Extended Request ID: OGWbDa4Lv6wwE9mBU6+QV+cYjZOgkRT3PFAvful1QZykKZ2t1ql/bNJaeiTZd/GjxFyE52/itms=; Proxy: null), S3 Extended Request ID: OGWbDa4Lv6wwE9mBU6+QV+cYjZOgkRT3PFAvful1QZykKZ2t1ql/bNJaeiTZd/GjxFyE52/itms=:400 Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: W01GKMAXC95KHVPR; S3 Extended Request ID: OGWbDa4Lv6wwE9mBU6+QV+cYjZOgkRT3PFAvful1QZykKZ2t1ql/bNJaeiTZd/GjxFyE52/itms=; Proxy: null) at org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.testJobSubmissionCollectsTokens(ITestDelegatedMRJob.java:281) Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: W01GKMAXC95KHVPR; S3 Extended Request ID: OGWbDa4Lv6wwE9mBU6+QV+cYjZOgkRT3PFAvful1QZykKZ2t1ql/bNJaeiTZd/GjxFyE52/itms=; Proxy: null) at org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob.testJobSubmissionCollectsTokens(ITestDelegatedMRJob.java:281) ``` > AWS AssumedRoleCredentialProvider needs ExternalId add > ------------------------------------------------------ > > Key: HADOOP-16806 > URL: https://issues.apache.org/jira/browse/HADOOP-16806 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.2.1 > Reporter: Jon Hartlaub > Priority: Minor > Labels: pull-request-available > > AWS has added a security feature to the assume-role function in the form of > the "ExternalId" key in the AWS Java SDK > {{STSAssumeRoleSessionCredentialsProvider.Builder}} class. To support this > security feature, the hadoop aws {{AssumedRoleCredentialProvider}} needs a > patch to include this value from the configuration as well as an added > Constant to the {{org.apache.hadoop.fs.s3a.Constants}} file. > The ExternalId is not a required security feature, it is an augmentation of > the current assume role configuration. > Proposed: > * Get the assume-role ExternalId token from the configuration for the > configuration key {{fs.s3a.assumed.role.externalid}} > * Use the configured ExternalId value in the > {{STSAssumeRoleSessionCredentialsProvider.Builder}} > e.g. > {{if (StringUtils.isNotEmpty(externalId)) {}} > {{ builder.withExternalId(externalId); // include the token for > cross-account assume role}} > {{}}} > Tests: > * +Unit test+ which verifies the ExternalId state value of the > {{AssumedRoleCredentialProvider}} is consistent with the configured value - > either empty or populated > * Question: not sure about how to write the +integration test+ for this > feature. We have an account configured for this use-case that verifies this > feature but I don't have much context on the Hadoop project AWS S3 > integration tests, perhaps a pointer could help. > > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org