[
https://issues.apache.org/jira/browse/HADOOP-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17703711#comment-17703711
]
ASF GitHub Bot commented on HADOOP-18565:
-----------------------------------------
ahmarsuhail commented on PR #5421:
URL: https://github.com/apache/hadoop/pull/5421#issuecomment-1479763570
I have some concerns for about this region first approach for third party
stores as well. The region probe is not going to work. I don't know how regions
actually work for third party stores, does the value of the region matter? If
no, set region to any valid region, otherwise third party store users must
always set `fs.s3a.endpoint.region`.
This kind of ties into a previous discussion, about detecting if the store
is AWS S3 or third party. I looked into it, and it's difficult to infer if a
store is AWS S3 or not reliably from the existing config values. Was thinking
we could do something like `Endpoint is not set || Endpoint is set && ends in
amazonaws.com || ARN is set ` but pretty sure this will miss edge cases, and
will be hard to maintain with new features etc. Maybe can add in a new config,
something like `fs.s3a.is.AWS` instead.
> AWS SDK V2 - Complete outstanding items
> ---------------------------------------
>
> Key: HADOOP-18565
> URL: https://issues.apache.org/jira/browse/HADOOP-18565
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Ahmar Suhail
> Priority: Major
> Labels: pull-request-available
>
> The following work remains to complete the SDK upgrade work:
> * S3A allows users configure to custom signers, add in support for this.
> * Remove SDK V1 bundle dependency
> * Update `getRegion()` logic to use retries.
> * Add in progress listeners for `S3ABlockOutputStream`
> * Fix any failing tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
