[
https://issues.apache.org/jira/browse/HADOOP-18938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812381#comment-17812381
]
ASF GitHub Bot commented on HADOOP-18938:
-----------------------------------------
steveloughran commented on code in PR #6466:
URL: https://github.com/apache/hadoop/pull/6466#discussion_r1471473061
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java:
##########
@@ -361,6 +366,13 @@ private static URI getS3Endpoint(String endpoint, final
Configuration conf) {
*/
private static Region getS3RegionFromEndpoint(String endpoint) {
+ // S3 VPC endpoint parsing
+ Matcher matcher = VPC_ENDPOINT_PATTERN.matcher(endpoint);
+ if(matcher.find()) {
+ LOG.debug("Endpoint {} is vpc endpoint; parsing", endpoint);
+ return Region.of(matcher.group(1));
Review Comment:
could you log the group(1) value in the debug log so we can see exactly
which one it is. region and endpoint bindings are the cause of many of our
support calls.
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java:
##########
@@ -361,7 +366,15 @@ private static URI getS3Endpoint(String endpoint, final
Configuration conf) {
*/
private static Region getS3RegionFromEndpoint(String endpoint) {
Review Comment:
this could be made package private/visible for testing and then have some
unit tests which will run every time yetus does its work. this should include a
test which passes an endpoint which shouldn't match the regexp -and verifies
that it is rejected
> S3A region logic to handle vpce and non standard endpoints
> -----------------------------------------------------------
>
> Key: HADOOP-18938
> URL: https://issues.apache.org/jira/browse/HADOOP-18938
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Ahmar Suhail
> Priority: Major
> Labels: pull-request-available
>
> For non standard endpoints such as VPCE the region parsing added in
> HADOOP-18908 doesn't work. This is expected as that logic is only meant to be
> used for standard endpoints.
> If you are using a non-standard endpoint, check if a region is also provided,
> else fail fast.
> Also update documentation to explain to region and endpoint behaviour with
> SDK V2.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
