ahmarsuhail opened a new pull request, #5421: URL: https://github.com/apache/hadoop/pull/5421
### Description of PR Jira: https://issues.apache.org/jira/browse/HADOOP-18565 This PR: - Removes TM dependency, as it’s in the bundle now - Removes SDK V1 bundle dependency, only need `aws-java-sdk-core` now - Fixes` SIGNING_ALGORITHM_STS` identifier, I don’t think this was getting picked up correctly until now. - Removes V1 client creation and config code. CSE code has been removed too, will be added back in once S3EC becomes available in V2. - Adds retries to `getRegion()` logic. Also, if no endpoint is specified, don’t set the default endpoint. Let the SDK figure it out from the region. - You can’t currently add progress listeners to individual operations in the SDK. So modify existing progress listeners for MPU and putObject in `S3ABlockOutputStream`. - Updates reflection methods in `S3AUtils` to use generics, so it can be used by V1 Cred provider, V2 cred provider, and signers. - Adds in custom signer support. No signer factory in V2, so create our own. - CopyEncryptionParams logic - Use src object settings if present, otherwise encrypt using FS settings. - ITestS3AEndpointRegion - There’s no client.getRegionName(), so to get these tests to work, have used execution interceptors that throw an exception to exit early. - Removes some tests in TestNetworkBinding, looks like they were getting skipped anyway. - Removes ITestMarkerTool.testRunWrongBucket - I don’t have a good understanding of ITestMarkerTool, but I believe with the new `getRegion()` logic, which will throw an `UnknownStoreException`, we no longer need this test. It currently fails, because it tries to intialise a FS for a bucket which does not exist. ### Still TODO - Cache regions per FS to prevent multiple head bucket calls, warn if no region is configured. - Look at probing logic, and how it can be optimised. As a headBucket call may already have been made, we can avoid another probe. - GetObjectMetadata() will fail for base path ("/") (empty key), is this ok? ### How was this patch tested? Tested in `eu-west-1` by running mvn -Dparallel-tests -DtestsThreadCount=16 clean verify. All tests pass now. Note, to get all tests to pass, we should now configure our config with the bucket specific region. Eg: ``` <property> <name>fs.s3a.bucket.<my-test-bucket>.endpoint.region</name> <value>eu-west-1</value> </property> ``` This is so tests that rely on public buckets don't fail. They will do a bucket probe to get the bucket region and configure the region. If the region was set using `fs.s3a.endpoint.region`, tests which use public buckets will also end up using this region, and since there is no cross region support, they end up failing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
