[
https://issues.apache.org/jira/browse/HADOOP-17705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17622232#comment-17622232
]
Greg Senia commented on HADOOP-17705:
-------------------------------------
[~ste...@apache.org] exactly it fixes it completely!! Not sure why they (AWS)
didn't fix this but makes me think they don't have very good test cases to test
V4 Sig and VPC endpoints as he added one to the test ..
As Cooper Walbrun put it in his PR:
I believe that requests from this SDK to S3 via VPC endpoints are currently
receiving 400 (Bad Request) responses. When I inspect the request headers being
sent in by the SDK, I noticed a difference in the Authorization header:
Authorization: AWS4-HMAC-SHA256
Credential=ACCESS_KEY_ID/20210325/vpce/s3/aws4_request # VPC endpoint
Authorization: AWS4-HMAC-SHA256
Credential=ACCESS_KEY_ID/20210325/us-east-2/s3/aws4_request # normal
The Credential value was being assembled using what the SDK misunderstood as
the region in VPC endpoints, which have the structure
vpce-xxxxxxxxxxxxxxxxx-xxxxxxxx.s3.[region].vpce.amazonaws.com in the case of
S3.
Upon further inspection of the code, it seems the issue originates from the
AwsHostNameUtils class, as you see in the diff of this pull request. I am
however new to this codebase so if I am incorrectly addressing this issue, let
me know. For what it is worth, I did confirm that my test in
AWS4SignerTest.java fails in the absence of my change in AwsHostNameUtils.java
> S3A to add option fs.s3a.endpoint.region to set AWS region
> ----------------------------------------------------------
>
> Key: HADOOP-17705
> URL: https://issues.apache.org/jira/browse/HADOOP-17705
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Mehakmeet Singh
> Assignee: Mehakmeet Singh
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently, AWS region is either constructed via the endpoint URL, by making
> an assumption that the 2nd component after delimiter "." is the region in
> endpoint URL, which doesn't work for private links and sets the default to
> us-east-1 thus causing authorization issue w.r.t the private link.
> The option fs.s3a.endpoint.region allows this to be explicitly set
> h2. how to set the s3 region on older hadoop releases
> For anyone who needs to set the signing region on older versions of the s3a
> client *you do not need this festure*. instead just provide a custom endpoint
> to region mapping json file
> # Download the default region mapping file
> [awssdk_config_default.json|https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/internal/config/awssdk_config_default.json]
> # Add a new regular expression to map the endpoint/hostname to the target
> region
> # Save the file as {{/etc/hadoop/conf/awssdk_config_override.json}}
> # verify basic hadop fs -ls commands work
> # copy to the rest of the cluster.
> # There should be no need to restart any services
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
