[jira] [Commented] (HADOOP-18853) AWS SDK V2 - Integrate new transfer manager
[ https://issues.apache.org/jira/browse/HADOOP-18853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17757365#comment-17757365 ] ASF GitHub Bot commented on HADOOP-18853: - steveloughran commented on PR #5960: URL: https://github.com/apache/hadoop/pull/5960#issuecomment-1687943481 @ahmarsuhail can you rebase and retest this, then i'll merge. once in we can rebase the whole feature branch, retest -and then merge to trunk! > AWS SDK V2 - Integrate new transfer manager > --- > > Key: HADOOP-18853 > URL: https://issues.apache.org/jira/browse/HADOOP-18853 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Priority: Major > Labels: pull-request-available > > With 2.20.121, the TM has MPU functionality. Upgrading to to this version > will also solve the issue with needing to include the CRT dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18853) AWS SDK V2 - Integrate new transfer manager
[
https://issues.apache.org/jira/browse/HADOOP-18853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755985#comment-17755985
]
ASF GitHub Bot commented on HADOOP-18853:
-
hadoop-yetus commented on PR #5960:
URL: https://github.com/apache/hadoop/pull/5960#issuecomment-1683970534
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ feature-HADOOP-18073-s3a-sdk-upgrade Compile Tests _ |
| +0 :ok: | mvndep | 14m 15s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 59s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | compile | 17m 27s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 16m 28s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 4m 29s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | mvnsite | 1m 50s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | javadoc | 1m 34s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 30s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +0 :ok: | spotbugs | 0m 52s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 1m 21s |
[/branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5960/1/artifact/out/branch-spotbugs-hadoop-tools_hadoop-aws-warnings.html)
| hadoop-tools/hadoop-aws in feature-HADOOP-18073-s3a-sdk-upgrade has 1
extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 34m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 58s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 48s | | the patch passed |
| +1 :green_heart: | compile | 16m 48s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 16m 48s | | the patch passed |
| +1 :green_heart: | compile | 16m 7s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 16m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 40s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +0 :ok: | spotbugs | 0m 47s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 34m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 45s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 3m 11s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 14s | | The patch does not
generate ASF License warnings. |
| | | 219m 39s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5960/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5960 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux c70e7301253f
[jira] [Commented] (HADOOP-18853) AWS SDK V2 - Integrate new transfer manager
[ https://issues.apache.org/jira/browse/HADOOP-18853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755965#comment-17755965 ] ASF GitHub Bot commented on HADOOP-18853: - steveloughran commented on code in PR #5960: URL: https://github.com/apache/hadoop/pull/5960#discussion_r1298431726 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java: ## @@ -98,17 +99,25 @@ public S3AsyncClient createS3AsyncClient( Configuration conf = getConf(); String bucket = uri.getHost(); + NettyNioAsyncHttpClient.Builder httpClientBuilder = AWSClientConfig Review Comment: is this going to build properly with unshaded artifacts? we've caused problems in the path (#2599) because of refs to .shaded classes.. the netty and client stuff are public/stable unshaded classes, correct? > AWS SDK V2 - Integrate new transfer manager > --- > > Key: HADOOP-18853 > URL: https://issues.apache.org/jira/browse/HADOOP-18853 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Priority: Major > Labels: pull-request-available > > With 2.20.121, the TM has MPU functionality. Upgrading to to this version > will also solve the issue with needing to include the CRT dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18853) AWS SDK V2 - Integrate new transfer manager
[ https://issues.apache.org/jira/browse/HADOOP-18853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755919#comment-17755919 ] ASF GitHub Bot commented on HADOOP-18853: - ahmarsuhail commented on PR #5960: URL: https://github.com/apache/hadoop/pull/5960#issuecomment-1683736580 @steveloughran this updates the SDK version, so the Java async client has MPU again. Since Multipart operations are in the java async client now, and not the TM, we could consider removing the TM in a follow up PR. We'd lose out on the transfer listener..but we don't really use any of the other additional functionality the TM provides us. > AWS SDK V2 - Integrate new transfer manager > --- > > Key: HADOOP-18853 > URL: https://issues.apache.org/jira/browse/HADOOP-18853 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Priority: Major > Labels: pull-request-available > > With 2.20.121, the TM has MPU functionality. Upgrading to to this version > will also solve the issue with needing to include the CRT dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18853) AWS SDK V2 - Integrate new transfer manager
[ https://issues.apache.org/jira/browse/HADOOP-18853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17755901#comment-17755901 ] ASF GitHub Bot commented on HADOOP-18853: - ahmarsuhail opened a new pull request, #5960: URL: https://github.com/apache/hadoop/pull/5960 ### Description of PR * Upgrades SDK Version so it has the new Java Async client with MPU support. * Configures java async client with MPU threshold and part size * Removes CRT dependency Since Multipart operations are in the java async client now, and not the TM, we could consider removing the TM in a follow up PR. We'd lose out on the transfer listener..but we don't really use any of the other additional functionality the TM provides us. ### How was this patch tested? Testing in progress in eu-west-1 with `mvn -Dparallel-tests -DtestsThreadCount=16 clean verify` Also ran checked output of ITestS3HugeFileArrayBlocks.test_100_renameHugeFile(), time taken to rename a 256MB file is at par with V1, around ~1s from my m4.2xlarge EC2. > AWS SDK V2 - Integrate new transfer manager > --- > > Key: HADOOP-18853 > URL: https://issues.apache.org/jira/browse/HADOOP-18853 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Ahmar Suhail >Priority: Major > > With 2.20.121, the TM has MPU functionality. Upgrading to to this version > will also solve the issue with needing to include the CRT dependency. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
