[
https://issues.apache.org/jira/browse/HADOOP-19490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17934617#comment-17934617
]
ASF GitHub Bot commented on HADOOP-19490:
-----------------------------------------
steveloughran commented on PR #7494:
URL: https://github.com/apache/hadoop/pull/7494#issuecomment-2718762118
@jojochuang Thanks for looking for this...just reviewed the code
We close them in
`org.apache.hadoop.fs.s3a.impl.ClientManagerImpl.serviceStop()`; that does on
demand creation and parallel closing of the clients...the use of the async adds
overhead in startup/shutdown time without this (#6892). Any app which doesn't
cache fs instances across tasks executed for the same user still takes a hit
though
> S3A: AWS SDK 2.30+ incompatible with third party stores
> -------------------------------------------------------
>
> Key: HADOOP-19490
> URL: https://issues.apache.org/jira/browse/HADOOP-19490
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Steve Loughran
> Priority: Major
> Labels: pull-request-available
>
> Changes in the SDK related to a new AWS SDK Feature, Default integritY
> Protection
> {{https://docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html}}
> break all intereraction with third party S3 Stores.
> See {{https://github.com/aws/aws-sdk-java-v2/issues/5801}}
> There are documented mechanisms to turn this off
> * an environment variable
> * a change in ~/..aws/config
> * a system property
> * looks like a new builder option, though the docs don't cover it.
> That checksum builder option looks like the only viable strategy, but it will
> need testing. maybe even make it a property which can be enabled/disabled,
> with tests and docs.
> Whoever wants a 2.30.x feature either gets to do the fixing and testing. For
> 3.4.2 it'll be an older release.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
