[
https://issues.apache.org/jira/browse/HADOOP-17705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17620629#comment-17620629
]
ASF GitHub Bot commented on HADOOP-17705:
-----------------------------------------
gss2002 opened a new pull request, #5046:
URL: https://github.com/apache/hadoop/pull/5046
### Description of PR
Currently, AWS region is either constructed via the endpoint URL, by making
an assumption that the 2nd component after delimiter "." is the region in
endpoint URL, which doesn't work for private links and sets the default to
us-east-1 thus causing authorization issue w.r.t the private link.
The option fs.s3a.endpoint.region allows this to be explicitly set
### How was this patch tested?
~/apache-maven-3.8.6/bin/mvn integration-test -Dparallel-tests
-DtestsThreadCount=50 -Dscale -Ptests-on
[INFO] Scanning for projects...
[INFO]
------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO]
------------------------------------------------------------------------
[INFO] os.detected.name: linux
[INFO] os.detected.arch: x86_64
[INFO] os.detected.version: 4.18
[INFO] os.detected.version.major: 4
[INFO] os.detected.version.minor: 18
[INFO] os.detected.release: centos
[INFO] os.detected.release.version: 8
[INFO] os.detected.release.like.centos: true
[INFO] os.detected.release.like.rhel: true
[INFO] os.detected.release.like.fedora: true
[INFO] os.detected.classifier: linux-x86_64
[INFO]
[INFO] --------------------< org.apache.hadoop:hadoop-aws
>--------------------
[INFO] Building Apache Hadoop Amazon Web Services support 3.2.5-SNAPSHOT
[INFO] --------------------------------[ jar
]---------------------------------
[INFO]
[INFO] -
> S3A to add option fs.s3a.endpoint.region to set AWS region
> ----------------------------------------------------------
>
> Key: HADOOP-17705
> URL: https://issues.apache.org/jira/browse/HADOOP-17705
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Mehakmeet Singh
> Assignee: Mehakmeet Singh
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Currently, AWS region is either constructed via the endpoint URL, by making
> an assumption that the 2nd component after delimiter "." is the region in
> endpoint URL, which doesn't work for private links and sets the default to
> us-east-1 thus causing authorization issue w.r.t the private link.
> The option fs.s3a.endpoint.region allows this to be explicitly set
> h2. how to set the s3 region on older hadoop releases
> For anyone who needs to set the signing region on older versions of the s3a
> client *you do not need this festure*. instead just provide a custom endpoint
> to region mapping json file
> # Download the default region mapping file
> [awssdk_config_default.json|https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/internal/config/awssdk_config_default.json]
> # Add a new regular expression to map the endpoint/hostname to the target
> region
> # Save the file as {{/etc/hadoop/conf/awssdk_config_override.json}}
> # verify basic hadop fs -ls commands work
> # copy to the rest of the cluster.
> # There should be no need to restart any services
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
