[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[
https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17731132#comment-17731132
]
Viraj Jasani commented on HADOOP-18763:
---
we were excluding netty from aws-sdk?
{code:java}
com.amazonaws
aws-java-sdk-bundle
${aws-java-sdk.version}
io.netty
*
{code}
> Upgrade aws-java-sdk to 1.12.367+
> -
>
> Key: HADOOP-18763
> URL: https://issues.apache.org/jira/browse/HADOOP-18763
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.5
>Reporter: Steve Loughran
>Priority: Major
>
> aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is
> pulling in high severity CVE and creating unhappiness in security scans, even
> if s3a doesn't use that lib.
> The safe version for netty is netty:4.1.86.Final and this is used by
> aws-java-adk:1.12.367+
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17731508#comment-17731508 ] Steve Loughran commented on HADOOP-18763: - we were excluding netty from a shaded sdk which shouldnt declare a dependency on it. discussed in testing_s3a.md. the shaded artifacts are still in the big bundle jar > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Priority: Major > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17731659#comment-17731659 ] Wei-Chiu Chuang commented on HADOOP-18763: -- This is my first time touching the s3 connector. I managed to set up (following https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/testing.html#Qualifying_an_AWS_SDK_Update) and pass unit tests, integration tests, assumed roles, client side encryption. Did not run scale and load tests because I was using my peresonal aws account and unsure how much $$ bill it would rake in. > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Priority: Major > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17731662#comment-17731662 ] Viraj Jasani commented on HADOOP-18763: --- [~weichiu] i can help run full test suite with various options if you would like, i anyways run tests on a regular basis. > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Priority: Major > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17731668#comment-17731668 ] Wei-Chiu Chuang commented on HADOOP-18763: -- That would be really helpful. I don't think I'll be able to spend time on the cloud connectors. > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Priority: Major > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17731672#comment-17731672 ] Viraj Jasani commented on HADOOP-18763: --- sure thing, let me test 1.12.367 version today. can perform some manual testing and then do full test run with combination of scale and prefetch profiles. first, i can make it with trunk and once results are good, can repeat the same tests for 3.3. > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Priority: Major > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[
https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17731766#comment-17731766
]
Viraj Jasani commented on HADOOP-18763:
---
us-west-2:
mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dprefetch
errors so far:
{code:java}
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
1,920.089 s <<< FAILURE! - in
org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps
[ERROR]
testParallelRename(org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps) Time
elapsed: 960.003 s <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 96
milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:537)
at
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:88)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.putObject(S3ABlockOutputStream.java:628)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:428)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
at
org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps.parallelRenames(ITestS3AConcurrentOps.java:112)
at
org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps.testParallelRename(ITestS3AConcurrentOps.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
[ERROR]
testThreadPoolCoolDown(org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps)
Time elapsed: 960.005 s <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 96
milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:537)
at
org.apache.hadoop.thirdparty.com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:88)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.putObject(S3ABlockOutputStream.java:628)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:428)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
at
org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps.parallelRenames(ITestS3AConcurrentOps.java:112)
at
org.apache.hadoop.fs.s3a.scale.ITestS3AConcurrentOps.testThreadPoolCoolDown(ITestS3AConcurrentOps.java:189)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.ja
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17731782#comment-17731782 ] Viraj Jasani commented on HADOOP-18763: --- mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale results are quite good, no test failures (except for known failure of testRecursiveRootListing, which passes when run individually) > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Priority: Major > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[
https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17731791#comment-17731791
]
Viraj Jasani commented on HADOOP-18763:
---
this time, without vpn, all tests passed for prefetch profile as well (previous
failures testParallelRename and testThreadPoolCoolDown are no longer showing up
with full test run)
{code:java}
mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dprefetch {code}
> Upgrade aws-java-sdk to 1.12.367+
> -
>
> Key: HADOOP-18763
> URL: https://issues.apache.org/jira/browse/HADOOP-18763
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.5
>Reporter: Steve Loughran
>Priority: Major
>
> aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is
> pulling in high severity CVE and creating unhappiness in security scans, even
> if s3a doesn't use that lib.
> The safe version for netty is netty:4.1.86.Final and this is used by
> aws-java-adk:1.12.367+
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17731808#comment-17731808 ] ASF GitHub Bot commented on HADOOP-18763: - virajjasani opened a new pull request, #5741: URL: https://github.com/apache/hadoop/pull/5741 Jira: HADOOP-18763 > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Priority: Major > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17731809#comment-17731809 ] ASF GitHub Bot commented on HADOOP-18763: - virajjasani commented on PR #5741: URL: https://github.com/apache/hadoop/pull/5741#issuecomment-1588247976 `us-west-2`: two rounds of testing, results look good (details on Jira) ``` $ mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dprefetch ``` ``` $ mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale ``` > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Priority: Major > Labels: pull-request-available > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17731811#comment-17731811 ] ASF GitHub Bot commented on HADOOP-18763: - virajjasani commented on PR #5741: URL: https://github.com/apache/hadoop/pull/5741#issuecomment-1588254555 netty version: `4.1.86.Final` https://github.com/aws/aws-sdk-java/blob/1.12.367/pom.xml#L409-L410 > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Priority: Major > Labels: pull-request-available > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732152#comment-17732152 ] ASF GitHub Bot commented on HADOOP-18763: - jojochuang commented on PR #5741: URL: https://github.com/apache/hadoop/pull/5741#issuecomment-1589611728 Just from a code and functionality perspective LGTM. But folks like @steveloughran or @mukund-thakur or someone else more familiar with S3A need to chime in too. > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[
https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732182#comment-17732182
]
ASF GitHub Bot commented on HADOOP-18763:
-
hadoop-yetus commented on PR #5741:
URL: https://github.com/apache/hadoop/pull/5741#issuecomment-1589717056
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 21m 4s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 47s | | trunk passed |
| +1 :green_heart: | compile | 18m 37s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 17m 0s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | mvnsite | 21m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 9m 4s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 7m 27s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | shadedclient | 40m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 45s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 20m 33s | | the patch passed |
| +1 :green_heart: | compile | 17m 57s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 17m 57s | | the patch passed |
| +1 :green_heart: | compile | 16m 58s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 16m 58s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 15m 10s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 44s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 7m 33s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | shadedclient | 41m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 797m 20s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5741/1/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| -1 :x: | asflicense | 1m 39s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5741/1/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 1059m 40s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.mapreduce.v2.TestMRJobsWithProfiler |
| | hadoop.mapreduce.v2.TestMRJobs |
| | hadoop.mapreduce.v2.TestUberAM |
| |
hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 |
| | hadoop.yarn.client.TestFederationRMFailoverProxyProvider |
| | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5741/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5741 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs |
| uname | Linux e044c9d5c3d7 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3
19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git re
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732213#comment-17732213 ] ASF GitHub Bot commented on HADOOP-18763: - mukund-thakur commented on PR #5741: URL: https://github.com/apache/hadoop/pull/5741#issuecomment-1589846172 Was going through the comments on Jira. Looking good. Will run some tests myself. @virajjasani Some unit tests are failing in Yetus. Need to check those. This license issue seems unrelated to me `hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/yarnfederation/application_123456_0001/SC-1/_record:1:Missing Apache License` > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732226#comment-17732226 ] ASF GitHub Bot commented on HADOOP-18763: - jojochuang commented on PR #5741: URL: https://github.com/apache/hadoop/pull/5741#issuecomment-1589888266 Those failed tests don't rely on awsk sdk so can be ignored. > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732232#comment-17732232 ] ASF GitHub Bot commented on HADOOP-18763: - virajjasani commented on PR #5741: URL: https://github.com/apache/hadoop/pull/5741#issuecomment-1589899798 > Was going through the comments on Jira. Looking good. Will run some tests myself. > > @virajjasani Some unit tests are failing in Yetus. Need to check those. > > This license issue seems unrelated to me `hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/yarnfederation/application_123456_0001/SC-1/_record:1:Missing Apache License` yes sir, thanks for taking a look. i just confirmed that test majority failures are not relevant by comparing with various results from https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-trunk-java8-linux-x86_64 i was aware of mapreduce test failures as they appear all the time. moreover, TestObserverNode also seems to be appearing on some of the daily build results (hadoop-qbt-trunk-java8-linux-x86_64). > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732233#comment-17732233 ] ASF GitHub Bot commented on HADOOP-18763: - virajjasani commented on PR #5741: URL: https://github.com/apache/hadoop/pull/5741#issuecomment-1589901812 what i am not aware of is the failure on TestFederationRMFailoverProxyProvider, and also the asf warning. @goiri @slfan1989 could you please once take a look in case you are aware of them (just to make sure we can finally ignore them) > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732234#comment-17732234 ] ASF GitHub Bot commented on HADOOP-18763: - virajjasani commented on PR #5741: URL: https://github.com/apache/hadoop/pull/5741#issuecomment-1589903411 > Those failed tests don't rely on awsk sdk so can be ignored. agree, i will re-run the whole test suit against `us-west-2` again today. > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732265#comment-17732265 ] ASF GitHub Bot commented on HADOOP-18763: - virajjasani commented on PR #5741: URL: https://github.com/apache/hadoop/pull/5741#issuecomment-1590053507 another round looks good: `mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale -Dprefetch` > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732270#comment-17732270 ] ASF GitHub Bot commented on HADOOP-18763: - mukund-thakur commented on PR #5741: URL: https://github.com/apache/hadoop/pull/5741#issuecomment-1590087588 Running through most of the steps myself. Looking good. Just assumed role and access point tests are pending. > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732719#comment-17732719 ] ASF GitHub Bot commented on HADOOP-18763: - mukund-thakur merged PR #5741: URL: https://github.com/apache/hadoop/pull/5741 > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732724#comment-17732724 ] ASF GitHub Bot commented on HADOOP-18763: - jojochuang commented on PR #5741: URL: https://github.com/apache/hadoop/pull/5741#issuecomment-1591875609 Can we simply cherrypick this change into branch-3.3 or we need to do another round of testing? > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732729#comment-17732729 ] ASF GitHub Bot commented on HADOOP-18763: - mukund-thakur commented on PR #5741: URL: https://github.com/apache/hadoop/pull/5741#issuecomment-1591881436 I am just doing that. No around of testing not required. Just a full compilation is fine. > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18763) Upgrade aws-java-sdk to 1.12.367+
[ https://issues.apache.org/jira/browse/HADOOP-18763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732745#comment-17732745 ] ASF GitHub Bot commented on HADOOP-18763: - mukund-thakur commented on PR #5741: URL: https://github.com/apache/hadoop/pull/5741#issuecomment-1591906660 cp'ed locally, recompiled, and pushed to branch-3.3 > Upgrade aws-java-sdk to 1.12.367+ > - > > Key: HADOOP-18763 > URL: https://issues.apache.org/jira/browse/HADOOP-18763 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > aws sdk bundle < 1.12.367 uses a vulnerable versions of netty which is > pulling in high severity CVE and creating unhappiness in security scans, even > if s3a doesn't use that lib. > The safe version for netty is netty:4.1.86.Final and this is used by > aws-java-adk:1.12.367+ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
