[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648416&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648416
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 06:51
Start Date: 09/Sep/21 06:51
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915812602
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/14/console in
case of problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648416)
Time Spent: 10h 50m (was: 10h 40m)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 50m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
hadoop-yetus commented on pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915812602 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/14/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648415&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648415
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 06:45
Start Date: 09/Sep/21 06:45
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#discussion_r705017926
##
File path: hadoop-client-modules/hadoop-client-integration-tests/pom.xml
##
@@ -52,6 +52,16 @@
junit
test
+
Review comment:
I just tested it locally. Seems working. Let me change it and see CI
result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648415)
Time Spent: 10h 40m (was: 10.5h)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 40m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
viirya commented on a change in pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#discussion_r705017926 ## File path: hadoop-client-modules/hadoop-client-integration-tests/pom.xml ## @@ -52,6 +52,16 @@ junit test + Review comment: I just tested it locally. Seems working. Let me change it and see CI result. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17195) Intermittent OutOfMemory error while performing hdfs CopyFromLocal to abfs
[ https://issues.apache.org/jira/browse/HADOOP-17195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412379#comment-17412379 ] Mukund Thakur commented on HADOOP-17195: [~mehakmeetSingh] Could you please add the stacktrace in the description? > Intermittent OutOfMemory error while performing hdfs CopyFromLocal to abfs > --- > > Key: HADOOP-17195 > URL: https://issues.apache.org/jira/browse/HADOOP-17195 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.3.0 >Reporter: Mehakmeet Singh >Assignee: Mehakmeet Singh >Priority: Major > Labels: abfsactive, pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > OutOfMemory error due to new ThreadPools being made each time > AbfsOutputStream is created. Since threadPool aren't limited a lot of data is > loaded in buffer and thus it causes OutOfMemory error. > Possible fixes: > - Limit the number of ThreadCounts while performing hdfs copyFromLocal (Using > -t property). > - Reducing OUTPUT_BUFFER_SIZE significantly which would limit the amount of > buffer to be loaded in threads. > - Don't create new ThreadPools each time AbfsOutputStream is created and > limit the number of ThreadPools each AbfsOutputStream could create. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3389: YARN-10928. Support default queue config for minimum-user-limit-percent/user-limit-factor
hadoop-yetus commented on pull request #3389:
URL: https://github.com/apache/hadoop/pull/3389#issuecomment-915805998
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 54s | | trunk passed |
| +1 :green_heart: | compile | 10m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 8m 40s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 34s | |
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 14m 28s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 50s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 7s | | the patch passed |
| +1 :green_heart: | compile | 9m 26s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 9m 26s | | the patch passed |
| +1 :green_heart: | compile | 8m 42s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 8m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 35s | |
hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 107 unchanged - 1
fixed = 107 total (was 108) |
| +1 :green_heart: | mvnsite | 1m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 29s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 14m 11s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 96m 15s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 0m 38s | | hadoop-yarn-site in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 55s | | The patch does not
generate ASF License warnings. |
| | | 222m 49s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3389/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3389 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 4498ab1a4b2c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / dac8df06e79a91d3503b34ee7d0ea6da8f63855b |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3389/3/testReport/ |
| Max. process+thread count | 943 (vs. ulim
[jira] [Commented] (HADOOP-17854) Run junit in Jenkins only if surefire reports exist
[
https://issues.apache.org/jira/browse/HADOOP-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412377#comment-17412377
]
Ayush Saxena commented on HADOOP-17854:
---
Is this problem solved?
I still see the same exception message as in the image attached in all the PR.
eg:
[https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3406/1/pipeline]
{noformat}
[2021-09-08T19:52:17.078Z] No test report files were found. Configuration
error?{noformat}
> Run junit in Jenkins only if surefire reports exist
> ---
>
> Key: HADOOP-17854
> URL: https://issues.apache.org/jira/browse/HADOOP-17854
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: image-2021-08-18-08-59-14-022.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> We need to check if some xml files exist under surefire-reports before
> running junit in Jenkins.
> !image-2021-08-18-08-59-14-022.png|thumbnail!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomicooler commented on a change in pull request #3355: YARN-10901. Permission checking error on an existing directory in LogAggregationFileController#verifyAndCreateRemoteLogDir
tomicooler commented on a change in pull request #3355:
URL: https://github.com/apache/hadoop/pull/3355#discussion_r705011903
##
File path:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileController.java
##
@@ -427,17 +428,25 @@ public void verifyAndCreateRemoteLogDir() {
throw new YarnRuntimeException("Failed to create remoteLogDir ["
+ remoteRootLogDir + "]", e);
}
-} else{
+} else {
//Check if FS has capability to set/modify permissions
+ Path permissionCheckFile = new Path(qualified,
String.format("%s.permission_check",
+ RandomStringUtils.randomAlphanumeric(8)));
try {
-remoteFS.setPermission(qualified, new FsPermission(TLDIR_PERMISSIONS));
+remoteFS.createNewFile(permissionCheckFile);
+remoteFS.setPermission(permissionCheckFile, new
FsPermission(TLDIR_PERMISSIONS));
} catch (UnsupportedOperationException use) {
LOG.info("Unable to set permissions for configured filesystem since"
Review comment:
https://issues.apache.org/jira/browse/YARN-10937
##
File path:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/LogAggregationFileController.java
##
@@ -427,17 +428,25 @@ public void verifyAndCreateRemoteLogDir() {
throw new YarnRuntimeException("Failed to create remoteLogDir ["
+ remoteRootLogDir + "]", e);
}
-} else{
+} else {
//Check if FS has capability to set/modify permissions
+ Path permissionCheckFile = new Path(qualified,
String.format("%s.permission_check",
+ RandomStringUtils.randomAlphanumeric(8)));
try {
-remoteFS.setPermission(qualified, new FsPermission(TLDIR_PERMISSIONS));
+remoteFS.createNewFile(permissionCheckFile);
+remoteFS.setPermission(permissionCheckFile, new
FsPermission(TLDIR_PERMISSIONS));
} catch (UnsupportedOperationException use) {
LOG.info("Unable to set permissions for configured filesystem since"
+ " it does not support this", remoteFS.getScheme());
fsSupportsChmod = false;
} catch (IOException e) {
LOG.warn("Failed to check if FileSystem suppports permissions on "
Review comment:
https://issues.apache.org/jira/browse/YARN-10936
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3389: YARN-10928. Support default queue config for minimum-user-limit-percent/user-limit-factor
hadoop-yetus commented on pull request #3389:
URL: https://github.com/apache/hadoop/pull/3389#issuecomment-915799537
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 39s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 31s | | trunk passed |
| +1 :green_heart: | compile | 10m 11s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 8m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 23s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 31s | |
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 15m 56s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 16m 17s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 6s | | the patch passed |
| +1 :green_heart: | compile | 8m 58s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 8m 58s | | the patch passed |
| +1 :green_heart: | compile | 8m 28s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 8m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 33s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3389/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 5 new + 107 unchanged
- 1 fixed = 112 total (was 108) |
| +1 :green_heart: | mvnsite | 1m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 26s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 14m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 96m 9s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 0m 50s | | hadoop-yarn-site in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 223m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3389/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3389 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux b5eb986c2976 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 96ca2ac2689bfd0f2c7ad76d13351693612dee5b |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648405&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648405
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 06:17
Start Date: 09/Sep/21 06:17
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#discussion_r705000606
##
File path: hadoop-client-modules/hadoop-client-integration-tests/pom.xml
##
@@ -52,6 +52,16 @@
junit
test
+
Review comment:
Oh I see. It makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648405)
Time Spent: 10.5h (was: 10h 20m)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomicooler commented on a change in pull request #3390: YARN-10912. Refactor AbstractCSQueue#updateConfigurableResourceRequir…
tomicooler commented on a change in pull request #3390:
URL: https://github.com/apache/hadoop/pull/3390#discussion_r705000607
##
File path:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
##
@@ -621,39 +619,35 @@ protected void
updateConfigurableResourceRequirement(String queuePath,
} else {
validateAbsoluteVsPercentageCapacityConfig(localType);
}
+}
+ }
- // If min resource for a resource type is greater than its max resource,
- // throw exception to handle such error configs.
- if (!maxResource.equals(Resources.none()) && Resources.greaterThan(
- resourceCalculator, clusterResource, minResource, maxResource)) {
-throw new IllegalArgumentException("Min resource configuration "
-+ minResource + " is greater than its max value:" + maxResource
-+ " in queue:" + getQueuePath());
- }
+ protected void updateConfigurableResourceRequirement(Resource
clusterResource) {
Review comment:
Done, renamed it to: updateConfigurableResourceLimits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
viirya commented on a change in pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#discussion_r705000606 ## File path: hadoop-client-modules/hadoop-client-integration-tests/pom.xml ## @@ -52,6 +52,16 @@ junit test + Review comment: Oh I see. It makes sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2773: Test PR. Igonre.
hadoop-yetus commented on pull request #2773:
URL: https://github.com/apache/hadoop/pull/2773#issuecomment-915794250
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 28s | | trunk passed |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 0m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 26s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 0m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 20m 47s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 91m 2s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2773/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2773 |
| Optional Tests | dupname asflicense codespell shellcheck shelldocs compile
javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle |
| uname | Linux 2a9be539 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8ab7f527f6c6939785793afc87e87ea64e1d711f |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2773/5/testReport/ |
| Max. process+thread count | 544 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2773/5/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-un
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648403&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648403
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 06:10
Start Date: 09/Sep/21 06:10
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#discussion_r704994380
##
File path: hadoop-client-modules/hadoop-client-integration-tests/pom.xml
##
@@ -57,6 +57,11 @@
lz4-java
test
+
+ org.xerial.snappy
+ snappy-java
+ test
Review comment:
Because we exclude it from the uber jar, we don't include the artifact
of snappy-java in client libraries like hadoop-client-runtime. Note that both
hadoop-client-runtime and hadoop-client-api make hadoop-client as optional. So
it won't count hadoop-client as transitive dependency of these client libraries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648403)
Time Spent: 10h 20m (was: 10h 10m)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 20m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
viirya commented on a change in pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#discussion_r704994380 ## File path: hadoop-client-modules/hadoop-client-integration-tests/pom.xml ## @@ -57,6 +57,11 @@ lz4-java test + + org.xerial.snappy + snappy-java + test Review comment: Because we exclude it from the uber jar, we don't include the artifact of snappy-java in client libraries like hadoop-client-runtime. Note that both hadoop-client-runtime and hadoop-client-api make hadoop-client as optional. So it won't count hadoop-client as transitive dependency of these client libraries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648400&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648400
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 06:04
Start Date: 09/Sep/21 06:04
Worklog Time Spent: 10m
Work Description: sunchao commented on a change in pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#discussion_r704988707
##
File path: hadoop-client-modules/hadoop-client-integration-tests/pom.xml
##
@@ -52,6 +52,16 @@
junit
test
+
Review comment:
This doesn't look like the right approach, since the integration tests
shouldn't depend on test dependency. As mentioned earlier, I think the issue is
that the `snappy-java` is included in the optional dependency `hadoop-client`
so it won't be a transitive dependency for downstream apps. Therefore, those
apps will have to explicitly declare `snappy-java` as a dependency, which could
cause surprises.
Instead I think we should add `snappy-java` as compile dependency in
`hadoop-client-api`. You may need to fight the maven enforcer rules for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648400)
Time Spent: 10h 10m (was: 10h)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 10m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
sunchao commented on a change in pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#discussion_r704988707 ## File path: hadoop-client-modules/hadoop-client-integration-tests/pom.xml ## @@ -52,6 +52,16 @@ junit test + Review comment: This doesn't look like the right approach, since the integration tests shouldn't depend on test dependency. As mentioned earlier, I think the issue is that the `snappy-java` is included in the optional dependency `hadoop-client` so it won't be a transitive dependency for downstream apps. Therefore, those apps will have to explicitly declare `snappy-java` as a dependency, which could cause surprises. Instead I think we should add `snappy-java` as compile dependency in `hadoop-client-api`. You may need to fight the maven enforcer rules for this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648399&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648399
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 06:03
Start Date: 09/Sep/21 06:03
Worklog Time Spent: 10m
Work Description: dbtsai commented on a change in pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#discussion_r704987732
##
File path: hadoop-client-modules/hadoop-client-integration-tests/pom.xml
##
@@ -57,6 +57,11 @@
lz4-java
test
+
+ org.xerial.snappy
+ snappy-java
+ test
Review comment:
If the hadoop client jar already depends on snappy-java transitively,
why do we need to add it to test scope?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648399)
Time Spent: 10h (was: 9h 50m)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] dbtsai commented on a change in pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
dbtsai commented on a change in pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#discussion_r704987732 ## File path: hadoop-client-modules/hadoop-client-integration-tests/pom.xml ## @@ -57,6 +57,11 @@ lz4-java test + + org.xerial.snappy + snappy-java + test Review comment: If the hadoop client jar already depends on snappy-java transitively, why do we need to add it to test scope? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17902) Fix Hadoop build on Debian 10
[
https://issues.apache.org/jira/browse/HADOOP-17902?focusedWorklogId=648398&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648398
]
ASF GitHub Bot logged work on HADOOP-17902:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 05:59
Start Date: 09/Sep/21 05:59
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#issuecomment-915786900
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/2/console in
case of problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648398)
Time Spent: 40m (was: 0.5h)
> Fix Hadoop build on Debian 10
> -
>
> Key: HADOOP-17902
> URL: https://issues.apache.org/jira/browse/HADOOP-17902
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> We're using *Debian testing* as one of the package sources to get the latest
> packages. It seems to be broken at the moment. The CI fails to create the
> build environment for the Debian 10 platform -
> {code}
> [2021-09-08T00:21:11.596Z] #13 [ 8/14] RUN apt-get -q update && apt-get
> -q install -y --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*
> ...
> [2021-09-08T00:21:22.744Z] #13 11.28 Preparing to unpack
> .../libc6_2.31-17_amd64.deb ...
> [2021-09-08T00:21:23.260Z] #13 11.46 Checking for services that may need to
> be restarted...
> [2021-09-08T00:21:23.260Z] #13 11.48 Checking init scripts...
> [2021-09-08T00:21:23.260Z] #13 11.50 Unpacking libc6:amd64 (2.31-17) over
> (2.28-10) ...
> [2021-09-08T00:21:26.290Z] #13 14.38 Setting up libc6:amd64 (2.31-17) ...
> [2021-09-08T00:21:26.290Z] #13 14.42 /usr/bin/perl: error while loading
> shared libraries: libcrypt.so.1: cannot open shared object file: No such file
> or directory
> [2021-09-08T00:21:26.290Z] #13 14.42 dpkg: error processing package
> libc6:amd64 (--configure):
> [2021-09-08T00:21:26.290Z] #13 14.42 installed libc6:amd64 package
> post-installation script subprocess returned error exit status 127
> [2021-09-08T00:21:26.291Z] #13 14.43 Errors were encountered while processing:
> [2021-09-08T00:21:26.291Z] #13 14.43 libc6:amd64
> [2021-09-08T00:21:26.291Z] #13 14.46 E: Sub-process /usr/bin/dpkg returned an
> error code (1)
> [2021-09-08T00:21:27.867Z] #13 ERROR: executor failed running [/bin/bash -o
> pipefail -c apt-get -q update && apt-get -q install -y
> --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*]: exit code: 100
> [2021-09-08T00:21:27.867Z] --
> [2021-09-08T00:21:27.867Z] > [ 8/14] RUN apt-get -q update && apt-get -q
> install -y --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*:
> [2021-09-08T00:21:27.867Z] --
> [2021-09-08T00:21:27.867Z] executor failed running [/bin/bash -o pipefail -c
> apt-get -q update && apt-get -q install -y --no-install-recommends
> python3 && apt-get -q install -y --no-install-recommends
> $(pkg-resolver/resolve.py debian:10) && apt-get clean && rm -rf
> /var/lib/apt/lists/*]: exit code: 100
> [2021-09-08T00:21:27.867Z] ERROR: Docker failed to build
> yetus/hadoop:ef5dbc7283a.
> [2021-09-08T00:21:27.867Z]
> {code}
> The above log lines are copied from -
> https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3388/3/pipeline
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3408: HADOOP-17902. Fix Hadoop build on Debian 10
hadoop-yetus commented on pull request #3408: URL: https://github.com/apache/hadoop/pull/3408#issuecomment-915786900 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/2/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17902) Fix Hadoop build on Debian 10
[
https://issues.apache.org/jira/browse/HADOOP-17902?focusedWorklogId=648397&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648397
]
ASF GitHub Bot logged work on HADOOP-17902:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 05:57
Start Date: 09/Sep/21 05:57
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#issuecomment-915786069
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 26m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 57s | | trunk passed |
| +1 :green_heart: | compile | 2m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 42m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 2m 30s | | the patch passed |
| +1 :green_heart: | cc | 2m 30s | | the patch passed |
| +1 :green_heart: | golang | 2m 30s | | the patch passed |
| +1 :green_heart: | javac | 2m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 108m 8s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 199m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3408 |
| Optional Tests | dupname asflicense codespell hadolint shellcheck
shelldocs compile cc mvnsite javac unit golang |
| uname | Linux 2d0cad2f6762 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13
01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / aeb5bfe5d24be52cb4e27e61f4ecc58bf1d7e2b0 |
| Default Java | Red Hat, Inc.-1.8.0_302-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/2/testReport/ |
| Max. process+thread count | 518 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/2/console |
| versions | git=2.27.0 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648397)
Time Spent: 0.5h (was: 20m)
> Fix Hadoop build on Debian 10
> -
>
> Key: HADOOP-17902
> URL: https://issues.apache.org/jira/browse/HADOOP-17902
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> We're using *Debian testing* as one of the packa
[GitHub] [hadoop] hadoop-yetus commented on pull request #3408: HADOOP-17902. Fix Hadoop build on Debian 10
hadoop-yetus commented on pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#issuecomment-915786069
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 26m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 57s | | trunk passed |
| +1 :green_heart: | compile | 2m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 42m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 2m 30s | | the patch passed |
| +1 :green_heart: | cc | 2m 30s | | the patch passed |
| +1 :green_heart: | golang | 2m 30s | | the patch passed |
| +1 :green_heart: | javac | 2m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 108m 8s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 199m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3408 |
| Optional Tests | dupname asflicense codespell hadolint shellcheck
shelldocs compile cc mvnsite javac unit golang |
| uname | Linux 2d0cad2f6762 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13
01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / aeb5bfe5d24be52cb4e27e61f4ecc58bf1d7e2b0 |
| Default Java | Red Hat, Inc.-1.8.0_302-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/2/testReport/ |
| Max. process+thread count | 518 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/2/console |
| versions | git=2.27.0 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant edited a comment on pull request #3340: HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
bshashikant edited a comment on pull request #3340: URL: https://github.com/apache/hadoop/pull/3340#issuecomment-915780582 > @bshashikant , thank you for the patch. > > I see that the patch works by converting logic from reference-equals to value-equals. In `AclStorage`, we maintain a reference-counted mapping of every immutable instance of every `AclFeature` currently in use by the namesystem. Inodes with identical ACL entries all point to the same `AclFeature`, so reference equals should work fine. This was an intentional choice of the ACL design to lower memory footprint and provide an inexpensive equality operation. > > Xattrs do not use this same flyweight pattern though, so I can see why deep equals would be necessary there. I wonder if this bug really only happens for xattrs and not ACLs? If the bug happens for ACLs, then that might mean there is really something wrong with reference counting in `AclStorage`, causing duplicate instances for the same ACL entries. > > Also, if value equals is necessary, then there is similar logic in `INodeFile` and `INodeFileAttributes`. Thanks @cnauroth for the explanation. I tested the acl feature and it seems the issue only seem to exist with Xattr feature not acl itself as you suggested. I have modified the patch accordingly. Thank you very much for the details. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on pull request #3340: HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
bshashikant commented on pull request #3340: URL: https://github.com/apache/hadoop/pull/3340#issuecomment-915780582 > @bshashikant , thank you for the patch. > > I see that the patch works by converting logic from reference-equals to value-equals. In `AclStorage`, we maintain a reference-counted mapping of every immutable instance of every `AclFeature` currently in use by the namesystem. Inodes with identical ACL entries all point to the same `AclFeature`, so reference equals should work fine. This was an intentional choice of the ACL design to lower memory footprint and provide an inexpensive equality operation. > > Xattrs do not use this same flyweight pattern though, so I can see why deep equals would be necessary there. I wonder if this bug really only happens for xattrs and not ACLs? If the bug happens for ACLs, then that might mean there is really something wrong with reference counting in `AclStorage`, causing duplicate instances for the same ACL entries. > > Also, if value equals is necessary, then there is similar logic in `INodeFile` and `INodeFileAttributes`. Thanks @cnauroth for the explanation. I tested the acl feature and it seems the issue only seem to exist with Xattr feature not acl itself as you suggested. I have modified the patch accordingly. Thank you very much for explanation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3360: YARN-10909. AbstractCSQueue: add @VisibleForTesting annotation for test code
hadoop-yetus commented on pull request #3360:
URL: https://github.com/apache/hadoop/pull/3360#issuecomment-915767370
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 37s | | trunk passed |
| +1 :green_heart: | compile | 1m 1s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 59s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 0m 46s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 38s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 20s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 99m 35s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 183m 21s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3360/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3360 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux c97a2869c059 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b2506d7db3adbe5be16d0f49f490dcd3ffa26f64 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3360/2/testReport/ |
| Max. process+thread count | 907 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3360/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648387&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648387
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 04:56
Start Date: 09/Sep/21 04:56
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915762366
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 18m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 35s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 0s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 58s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 32s | |
branch/hadoop-client-modules/hadoop-client-api no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output
file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-integration-tests no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 26m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 10m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 2s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-api has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-runtime has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
|
| +0 :ok: | spotbugs | 0m 17s | |
hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-integration-tests has no data from spotbugs
|
| +1 :green_heart: | shadedclient | 27m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-api in the pat
[GitHub] [hadoop] hadoop-yetus commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
hadoop-yetus commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915762366
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 18m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 35s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 0s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 58s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 32s | |
branch/hadoop-client-modules/hadoop-client-api no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output
file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-integration-tests no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 26m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 10m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 2s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-api has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-runtime has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
|
| +0 :ok: | spotbugs | 0m 17s | |
hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-integration-tests has no data from spotbugs
|
| +1 :green_heart: | shadedclient | 27m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-api in the patch
passed. |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-runtime in the
patch passed. |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-check-invariants
in the patch passed. |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-minicluster in
the patch passed. |
| +1 :green_heart: | unit | 0m 16s | | hadoop-client-integration-tests
in the patch passed. |
| +1 :green_
[jira] [Work logged] (HADOOP-17892) Add Hadoop code formatter in dev-support
[ https://issues.apache.org/jira/browse/HADOOP-17892?focusedWorklogId=648382&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648382 ] ASF GitHub Bot logged work on HADOOP-17892: --- Author: ASF GitHub Bot Created on: 09/Sep/21 04:41 Start Date: 09/Sep/21 04:41 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3387: URL: https://github.com/apache/hadoop/pull/3387#issuecomment-915757232 > this is a big enough change you should mention it on common dev I have initiated a thread on common-dev. Thanks @aajisaka @jojochuang @tasanuma could you please take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648382) Time Spent: 1h 40m (was: 1.5h) > Add Hadoop code formatter in dev-support > > > Key: HADOOP-17892 > URL: https://issues.apache.org/jira/browse/HADOOP-17892 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > We should add Hadoop code formatter xml to dev-support specifically for new > developers to refer to. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3387: HADOOP-17892. Add Hadoop code formatter in dev-support
virajjasani commented on pull request #3387: URL: https://github.com/apache/hadoop/pull/3387#issuecomment-915757232 > this is a big enough change you should mention it on common dev I have initiated a thread on common-dev. Thanks @aajisaka @jojochuang @tasanuma could you please take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17904) Test Result Not Working In Jenkins Result
Ayush Saxena created HADOOP-17904: - Summary: Test Result Not Working In Jenkins Result Key: HADOOP-17904 URL: https://issues.apache.org/jira/browse/HADOOP-17904 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.4.0 Reporter: Ayush Saxena The jenkins used to show the Test Results, now the list is broken and No PR has that working. eg: [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3404/1/testReport/] For Daily Build No Test Result as well: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/622/ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648381&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648381
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 04:28
Start Date: 09/Sep/21 04:28
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915569374
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 30s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 0s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 57s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 28s | |
branch/hadoop-client-modules/hadoop-client-api no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output
file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-integration-tests no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 26m 37s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 10m 1s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 14s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 17s | |
hadoop-client-modules/hadoop-client-api has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-runtime has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
|
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-integration-tests has no data from spotbugs
|
| +1 :green_heart: | shadedclient | 27m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-api in
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
hadoop-yetus removed a comment on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915569374
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 30s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 0s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 57s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 28s | |
branch/hadoop-client-modules/hadoop-client-api no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output
file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-integration-tests no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 26m 37s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 10m 1s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 14s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 17s | |
hadoop-client-modules/hadoop-client-api has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-runtime has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
|
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-integration-tests has no data from spotbugs
|
| +1 :green_heart: | shadedclient | 27m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-api in the patch
passed. |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-runtime in the
patch passed. |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-check-invariants
in the patch passed. |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-minicluster in
the patch passed. |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-integration-tests
in the patch passed. |
| +1
[jira] [Updated] (HADOOP-17825) Add BuiltInGzipCompressor
[ https://issues.apache.org/jira/browse/HADOOP-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HADOOP-17825: -- Component/s: common > Add BuiltInGzipCompressor > - > > Key: HADOOP-17825 > URL: https://issues.apache.org/jira/browse/HADOOP-17825 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 25h 20m > Remaining Estimate: 0h > > Currently, GzipCodec only supports BuiltInGzipDecompressor, if native zlib is > not loaded. So, without Hadoop native codec installed, saving SequenceFile > using GzipCodec will throw exception like "SequenceFile doesn't work with > GzipCodec without native-hadoop code!" > Same as other codecs which we migrated to using prepared packages (lz4, > snappy), it will be better if we support GzipCodec generally without Hadoop > native codec installed. Similar to BuiltInGzipDecompressor, we can use Java > Deflater to support BuiltInGzipCompressor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17877) BuiltInGzipCompressor header and trailer should not be static variables
[ https://issues.apache.org/jira/browse/HADOOP-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved HADOOP-17877. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Assignee: L. C. Hsieh Resolution: Fixed > BuiltInGzipCompressor header and trailer should not be static variables > --- > > Key: HADOOP-17877 > URL: https://issues.apache.org/jira/browse/HADOOP-17877 > Project: Hadoop Common > Issue Type: Bug >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h > Remaining Estimate: 0h > > In the newly added BuiltInGzipCompressor, we should not let header and > trailer as static variables as they are for different instances. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17887) Remove GzipOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-17887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun resolved HADOOP-17887. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Assignee: L. C. Hsieh Resolution: Fixed > Remove GzipOutputStream > --- > > Key: HADOOP-17887 > URL: https://issues.apache.org/jira/browse/HADOOP-17887 > Project: Hadoop Common > Issue Type: Improvement >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 4h > Remaining Estimate: 0h > > As we provide built-in gzip compressor, we can use it in compressor stream. > The wrapper GzipOutputStream can be removed now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17887) Remove GzipOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-17887?focusedWorklogId=648377&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648377 ] ASF GitHub Bot logged work on HADOOP-17887: --- Author: ASF GitHub Bot Created on: 09/Sep/21 04:23 Start Date: 09/Sep/21 04:23 Worklog Time Spent: 10m Work Description: sunchao commented on pull request #3377: URL: https://github.com/apache/hadoop/pull/3377#issuecomment-915751360 Merged. Thanks @viirya for the contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648377) Time Spent: 3h 50m (was: 3h 40m) > Remove GzipOutputStream > --- > > Key: HADOOP-17887 > URL: https://issues.apache.org/jira/browse/HADOOP-17887 > Project: Hadoop Common > Issue Type: Improvement >Reporter: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > As we provide built-in gzip compressor, we can use it in compressor stream. > The wrapper GzipOutputStream can be removed now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17887) Remove GzipOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-17887?focusedWorklogId=648376&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648376 ] ASF GitHub Bot logged work on HADOOP-17887: --- Author: ASF GitHub Bot Created on: 09/Sep/21 04:23 Start Date: 09/Sep/21 04:23 Worklog Time Spent: 10m Work Description: sunchao merged pull request #3377: URL: https://github.com/apache/hadoop/pull/3377 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648376) Time Spent: 3h 40m (was: 3.5h) > Remove GzipOutputStream > --- > > Key: HADOOP-17887 > URL: https://issues.apache.org/jira/browse/HADOOP-17887 > Project: Hadoop Common > Issue Type: Improvement >Reporter: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 3h 40m > Remaining Estimate: 0h > > As we provide built-in gzip compressor, we can use it in compressor stream. > The wrapper GzipOutputStream can be removed now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17887) Remove GzipOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-17887?focusedWorklogId=648378&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648378 ] ASF GitHub Bot logged work on HADOOP-17887: --- Author: ASF GitHub Bot Created on: 09/Sep/21 04:23 Start Date: 09/Sep/21 04:23 Worklog Time Spent: 10m Work Description: sunchao edited a comment on pull request #3377: URL: https://github.com/apache/hadoop/pull/3377#issuecomment-915751360 Merged to trunk. Thanks @viirya for the contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648378) Time Spent: 4h (was: 3h 50m) > Remove GzipOutputStream > --- > > Key: HADOOP-17887 > URL: https://issues.apache.org/jira/browse/HADOOP-17887 > Project: Hadoop Common > Issue Type: Improvement >Reporter: L. C. Hsieh >Priority: Major > Labels: pull-request-available > Time Spent: 4h > Remaining Estimate: 0h > > As we provide built-in gzip compressor, we can use it in compressor stream. > The wrapper GzipOutputStream can be removed now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao edited a comment on pull request #3377: HADOOP-17887. Remove the wrapper class GzipOutputStream
sunchao edited a comment on pull request #3377: URL: https://github.com/apache/hadoop/pull/3377#issuecomment-915751360 Merged to trunk. Thanks @viirya for the contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #3377: HADOOP-17887. Remove the wrapper class GzipOutputStream
sunchao commented on pull request #3377: URL: https://github.com/apache/hadoop/pull/3377#issuecomment-915751360 Merged. Thanks @viirya for the contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao merged pull request #3377: HADOOP-17887. Remove the wrapper class GzipOutputStream
sunchao merged pull request #3377: URL: https://github.com/apache/hadoop/pull/3377 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani edited a comment on pull request #3386: HDFS-16213. De-flake TestFsDatasetImpl#testDnRestartWithHardLink
virajjasani edited a comment on pull request #3386: URL: https://github.com/apache/hadoop/pull/3386#issuecomment-915750759 Thanks for spending some time @LeonGao91. Although I understand we can repro it only when we run twice locally but Jenkins does report this as failure sometimes in single run and hence, this is flaky already. Please check: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3359/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt https://user-images.githubusercontent.com/34790606/132622011-32d7c4bf-757d-4309-ac6d-ec4ef21b310e.png";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3386: HDFS-16213. De-flake TestFsDatasetImpl#testDnRestartWithHardLink
virajjasani commented on pull request #3386: URL: https://github.com/apache/hadoop/pull/3386#issuecomment-915750759 Thanks for spending some time @LeonGao91. Although I understand we can repro it only when we run twice locally but Jenkins does report this as failure sometimes. Please check: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3359/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt https://user-images.githubusercontent.com/34790606/132622011-32d7c4bf-757d-4309-ac6d-ec4ef21b310e.png";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] cnauroth merged pull request #3375: HDFS-16207. Remove NN logs stack trace for non-existent xattr query
cnauroth merged pull request #3375: URL: https://github.com/apache/hadoop/pull/3375 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a change in pull request #3386: HDFS-16213. De-flake TestFsDatasetImpl#testDnRestartWithHardLink
virajjasani commented on a change in pull request #3386:
URL: https://github.com/apache/hadoop/pull/3386#discussion_r704939307
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
##
@@ -1068,9 +1070,8 @@ static File moveBlockFiles(Block b, ReplicaInfo
replicaInfo, File destdir)
+ srcReplica + " metadata to "
+ dstMeta, e);
}
-if (LOG.isDebugEnabled()) {
- LOG.info("Linked " + srcReplica.getBlockURI() + " to " + dstFile);
-}
+LOG.info("Linked {} to {} . Dest meta file: {}",
Review comment:
This seems a bug, we are guarding with isDebugEnabled and if DEBUG is
enabled, we are logging at INFO level. Either we can use INFO or DEBUG directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] FrankinRUC commented on pull request #3389: YARN-10928. Support default queue config for minimum-user-limit-percent/user-limit-factor
FrankinRUC commented on pull request #3389: URL: https://github.com/apache/hadoop/pull/3389#issuecomment-915719352 Ok, already add new commit to fix them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648335&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648335
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 02:52
Start Date: 09/Sep/21 02:52
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915718437
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/13/console in
case of problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648335)
Time Spent: 9.5h (was: 9h 20m)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
hadoop-yetus commented on pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915718437 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/13/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17902) Fix Hadoop build on Debian 10
[
https://issues.apache.org/jira/browse/HADOOP-17902?focusedWorklogId=648331&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648331
]
ASF GitHub Bot logged work on HADOOP-17902:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 02:37
Start Date: 09/Sep/21 02:37
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#issuecomment-91571
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 26s | | trunk passed |
| +1 :green_heart: | compile | 2m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 53m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 13s | | the patch passed |
| +1 :green_heart: | compile | 2m 29s | | the patch passed |
| +1 :green_heart: | cc | 2m 29s | | the patch passed |
| +1 :green_heart: | golang | 2m 29s | | the patch passed |
| +1 :green_heart: | javac | 2m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 14s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 110m 17s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 186m 31s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3408 |
| Optional Tests | dupname asflicense codespell hadolint shellcheck
shelldocs compile cc mvnsite javac unit golang |
| uname | Linux 90a825a43c59 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13
01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / aeb5bfe5d24be52cb4e27e61f4ecc58bf1d7e2b0 |
| Default Java | Red Hat, Inc.-1.8.0_302-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/2/testReport/ |
| Max. process+thread count | 622 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/2/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648331)
Time Spent: 20m (was: 10m)
> Fix Hadoop build on Debian 10
> -
>
> Key: HADOOP-17902
> URL: https://issues.apache.org/jira/browse/HADOOP-17902
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We're using *Debian testing* as one of the package
[GitHub] [hadoop] hadoop-yetus commented on pull request #3408: HADOOP-17902. Fix Hadoop build on Debian 10
hadoop-yetus commented on pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#issuecomment-91571
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 26s | | trunk passed |
| +1 :green_heart: | compile | 2m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 53m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 13s | | the patch passed |
| +1 :green_heart: | compile | 2m 29s | | the patch passed |
| +1 :green_heart: | cc | 2m 29s | | the patch passed |
| +1 :green_heart: | golang | 2m 29s | | the patch passed |
| +1 :green_heart: | javac | 2m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 14s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 110m 17s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 186m 31s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3408 |
| Optional Tests | dupname asflicense codespell hadolint shellcheck
shelldocs compile cc mvnsite javac unit golang |
| uname | Linux 90a825a43c59 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13
01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / aeb5bfe5d24be52cb4e27e61f4ecc58bf1d7e2b0 |
| Default Java | Red Hat, Inc.-1.8.0_302-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/2/testReport/ |
| Max. process+thread count | 622 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/2/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648330&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648330
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 02:34
Start Date: 09/Sep/21 02:34
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915712161
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 53s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 16s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 58s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 30s | |
branch/hadoop-client-modules/hadoop-client-api no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 23s | |
branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output
file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-integration-tests no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 27m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 10m 9s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 2s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-api has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-runtime has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
|
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-integration-tests has no data from spotbugs
|
| -1 :x: | shadedclient | 27m 36s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-api in the patch
passed. |
[GitHub] [hadoop] hadoop-yetus commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
hadoop-yetus commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915712161
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 53s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 16s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 58s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 30s | |
branch/hadoop-client-modules/hadoop-client-api no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 23s | |
branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output
file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-integration-tests no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 27m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 10m 9s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 2s | | No new issues. |
| +1 :green_heart: | xml | 0m 6s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-api has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-runtime has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
|
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-integration-tests has no data from spotbugs
|
| -1 :x: | shadedclient | 27m 36s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-api in the patch
passed. |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-runtime in the
patch passed. |
| +1 :green_heart: | unit | 0m 18s | | hadoop-client-check-invariants
in the patch passed. |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-minicluster in
the patch passed. |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-integration-tests
in the patch passed. |
| +1 :green_heart: | asfl
[jira] [Work logged] (HADOOP-17804) Prometheus metrics only include the last set of labels
[
https://issues.apache.org/jira/browse/HADOOP-17804?focusedWorklogId=648328&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648328
]
ASF GitHub Bot logged work on HADOOP-17804:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 02:24
Start Date: 09/Sep/21 02:24
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3369:
URL: https://github.com/apache/hadoop/pull/3369#issuecomment-915708669
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 16s | | trunk passed |
| +1 :green_heart: | compile | 21m 21s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 39s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 20m 39s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 39s | | the patch passed |
| +1 :green_heart: | compile | 18m 36s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 7s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 5s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 44s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 13s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 179m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3369/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3369 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 066b2d06cc28 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e4fb6898e2f6ba4ec04cfaa3d76b7417135d4bce |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3369/7/testReport/ |
| Max. process+thread count | 1639 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3369/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
[GitHub] [hadoop] hadoop-yetus commented on pull request #3369: HADOOP-17804. Expose prometheus metrics only after a flush and dedupe with tag values
hadoop-yetus commented on pull request #3369:
URL: https://github.com/apache/hadoop/pull/3369#issuecomment-915708669
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 16s | | trunk passed |
| +1 :green_heart: | compile | 21m 21s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 39s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 20m 39s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 39s | | the patch passed |
| +1 :green_heart: | compile | 18m 36s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 7s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 5s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 44s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 13s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 179m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3369/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3369 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 066b2d06cc28 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e4fb6898e2f6ba4ec04cfaa3d76b7417135d4bce |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3369/7/testReport/ |
| Max. process+thread count | 1639 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3369/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---
[GitHub] [hadoop] JackWangCS commented on pull request #3360: YARN-10909. AbstractCSQueue: add @VisibleForTesting annotation for test code
JackWangCS commented on pull request #3360: URL: https://github.com/apache/hadoop/pull/3360#issuecomment-915701304 @brumi1024 I have changed the setLastSubmittedTimestamp to package private. Thanks for your review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] TaoYang526 commented on pull request #3389: YARN-10928. Support default queue config for minimum-user-limit-percent/user-limit-factor
TaoYang526 commented on pull request #3389: URL: https://github.com/apache/hadoop/pull/3389#issuecomment-915700830 @FrankinRUC There are 5 new checkstyle warnings and conflicts in CapacityScheduler.md, please take a look! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ferhui commented on pull request #3379: HDFS-16210. RBF: Add the option of refreshCallQueue to RouterAdmin
ferhui commented on pull request #3379: URL: https://github.com/apache/hadoop/pull/3379#issuecomment-915698438 @symious Thanks for contribution, @goiri Thanks for review! Merged to trunk! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ferhui merged pull request #3379: HDFS-16210. RBF: Add the option of refreshCallQueue to RouterAdmin
ferhui merged pull request #3379: URL: https://github.com/apache/hadoop/pull/3379 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648300&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648300
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 01:03
Start Date: 09/Sep/21 01:03
Worklog Time Spent: 10m
Work Description: viirya commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915678172
> Yea see that. Could you check if snappy-java will show up as transitive
dependency for hadoop-client-api?
The current dependency tree of `hadoop-client-api`:
```
[INFO] org.apache.hadoop:hadoop-client-api:jar:3.4.0-SNAPSHOT
[INFO] +- org.apache.hadoop:hadoop-client:jar:3.4.0-SNAPSHOT:compile
(optional)
[INFO] | +- org.apache.hadoop:hadoop-common:jar:3.4.0-SNAPSHOT:compile
(optional)
[INFO] | | +-
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.1.1:compile
(optional)
...
[INFO] | | +- dnsjava:dnsjava:jar:3.4.0:compile (optional)
[INFO] | | \- org.xerial.snappy:snappy-java:jar:1.1.8.2:compile (optional)
[INFO] | +- org.apache.hadoop:hadoop-hdfs-client:jar:3.4.0-SNAPSHOT:compile
(optional)
...
```
The current dependency tree of `hadoop-client-runtime`:
```
[INFO] org.apache.hadoop:hadoop-client-runtime:jar:3.4.0-SNAPSHOT
[INFO] +- org.apache.hadoop:hadoop-client:jar:3.4.0-SNAPSHOT:compile
(optional)
[INFO] | +- org.apache.hadoop:hadoop-common:jar:3.4.0-SNAPSHOT:compile
(optional)
[INFO] | | +-
org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.1.1:compile
(optional)
...
[INFO] | | +- dnsjava:dnsjava:jar:3.4.0:compile (optional)
[INFO] | | \- org.xerial.snappy:snappy-java:jar:1.1.8.2:compile (optional)
[INFO] | +- org.apache.hadoop:hadoop-hdfs-client:jar:3.4.0-SNAPSHOT:compile
(optional)
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648300)
Time Spent: 9h 10m (was: 9h)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 10m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:21
[GitHub] [hadoop] viirya commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
viirya commented on pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915678172 > Yea see that. Could you check if snappy-java will show up as transitive dependency for hadoop-client-api? The current dependency tree of `hadoop-client-api`: ``` [INFO] org.apache.hadoop:hadoop-client-api:jar:3.4.0-SNAPSHOT [INFO] +- org.apache.hadoop:hadoop-client:jar:3.4.0-SNAPSHOT:compile (optional) [INFO] | +- org.apache.hadoop:hadoop-common:jar:3.4.0-SNAPSHOT:compile (optional) [INFO] | | +- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.1.1:compile (optional) ... [INFO] | | +- dnsjava:dnsjava:jar:3.4.0:compile (optional) [INFO] | | \- org.xerial.snappy:snappy-java:jar:1.1.8.2:compile (optional) [INFO] | +- org.apache.hadoop:hadoop-hdfs-client:jar:3.4.0-SNAPSHOT:compile (optional) ... ``` The current dependency tree of `hadoop-client-runtime`: ``` [INFO] org.apache.hadoop:hadoop-client-runtime:jar:3.4.0-SNAPSHOT [INFO] +- org.apache.hadoop:hadoop-client:jar:3.4.0-SNAPSHOT:compile (optional) [INFO] | +- org.apache.hadoop:hadoop-common:jar:3.4.0-SNAPSHOT:compile (optional) [INFO] | | +- org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.1.1:compile (optional) ... [INFO] | | +- dnsjava:dnsjava:jar:3.4.0:compile (optional) [INFO] | | \- org.xerial.snappy:snappy-java:jar:1.1.8.2:compile (optional) [INFO] | +- org.apache.hadoop:hadoop-hdfs-client:jar:3.4.0-SNAPSHOT:compile (optional) ... ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648298&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648298
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 00:56
Start Date: 09/Sep/21 00:56
Worklog Time Spent: 10m
Work Description: sunchao commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915675636
Yea see that. Could you check if snappy-java will show up as transitive
dependency for hadoop-client-api?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648298)
Time Spent: 9h (was: 8h 50m)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
sunchao commented on pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915675636 Yea see that. Could you check if snappy-java will show up as transitive dependency for hadoop-client-api? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648294&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648294
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 00:50
Start Date: 09/Sep/21 00:50
Worklog Time Spent: 10m
Work Description: viirya commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915673627
In the latest commit, I excluded snappy-java from the (shaded) uber jar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648294)
Time Spent: 8h 50m (was: 8h 40m)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 50m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
viirya commented on pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915673627 In the latest commit, I excluded snappy-java from the (shaded) uber jar. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648285&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648285
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 00:35
Start Date: 09/Sep/21 00:35
Worklog Time Spent: 10m
Work Description: sunchao commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-91541
@viirya @dbtsai mentioned a good point that it's better to make the
`snappy-java` as a transitive dependency for downstream apps. Perhaps we can
exclude it from the uber jar and redeclare it as a transitive dependency (we
might need to exclude it from `Hadoop-client` since it is optional and declare
it again as a dependency).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648285)
Time Spent: 8h 40m (was: 8.5h)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648284&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648284
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 00:35
Start Date: 09/Sep/21 00:35
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915666346
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/12/console in
case of problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648284)
Time Spent: 8.5h (was: 8h 20m)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
sunchao commented on pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#issuecomment-91541 @viirya @dbtsai mentioned a good point that it's better to make the `snappy-java` as a transitive dependency for downstream apps. Perhaps we can exclude it from the uber jar and redeclare it as a transitive dependency (we might need to exclude it from `Hadoop-client` since it is optional and declare it again as a dependency). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
hadoop-yetus commented on pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915666346 (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/12/console in case of problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648283&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648283
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 00:34
Start Date: 09/Sep/21 00:34
Worklog Time Spent: 10m
Work Description: viirya commented on a change in pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#discussion_r704865424
##
File path:
hadoop-client-modules/hadoop-client-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh
##
@@ -67,6 +67,8 @@ allowed_expr+="|^krb5_udp-template.conf$"
# Jetty uses this style sheet for directory listings. TODO ensure our
# internal use of jetty disallows directory listings and remove this.
allowed_expr+="|^jetty-dir.css$"
+# Snappy java is native library. We cannot relocate it to under
org/apache/hadoop.
Review comment:
By default, JNI uses fully qualified class name to resolve native
methods. Alternatively, JNI provides an API `RegisterNatives` used to register
native methods that don't require to match with class name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648283)
Time Spent: 8h 20m (was: 8h 10m)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on a change in pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
viirya commented on a change in pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#discussion_r704865424 ## File path: hadoop-client-modules/hadoop-client-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh ## @@ -67,6 +67,8 @@ allowed_expr+="|^krb5_udp-template.conf$" # Jetty uses this style sheet for directory listings. TODO ensure our # internal use of jetty disallows directory listings and remove this. allowed_expr+="|^jetty-dir.css$" +# Snappy java is native library. We cannot relocate it to under org/apache/hadoop. Review comment: By default, JNI uses fully qualified class name to resolve native methods. Alternatively, JNI provides an API `RegisterNatives` used to register native methods that don't require to match with class name. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17843) Support IPV6 with IP for internal and external communication
[
https://issues.apache.org/jira/browse/HADOOP-17843?focusedWorklogId=648279&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648279
]
ASF GitHub Bot logged work on HADOOP-17843:
---
Author: ASF GitHub Bot
Created on: 09/Sep/21 00:14
Start Date: 09/Sep/21 00:14
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3290:
URL: https://github.com/apache/hadoop/pull/3290#issuecomment-915658156
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ HADOOP-17800 Compile Tests _ |
| +0 :ok: | mvndep | 11m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 57s | | HADOOP-17800 passed |
| +1 :green_heart: | compile | 23m 3s | | HADOOP-17800 passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 26s | | HADOOP-17800 passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 54s | | HADOOP-17800 passed |
| +1 :green_heart: | mvnsite | 2m 59s | | HADOOP-17800 passed |
| +1 :green_heart: | javadoc | 2m 28s | | HADOOP-17800 passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 57s | | HADOOP-17800 passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 0s | | HADOOP-17800 passed |
| +1 :green_heart: | shadedclient | 16m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 13s | | the patch passed |
| +1 :green_heart: | compile | 22m 8s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 8s | | the patch passed |
| +1 :green_heart: | compile | 19m 33s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 50s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3290/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 69 unchanged - 0 fixed = 70 total (was
69) |
| +1 :green_heart: | mvnsite | 3m 1s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 26s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 59s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 329m 36s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3290/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 0m 58s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3290/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
| hadoop-yarn-common in the patch failed. |
| +1 :green_heart: | unit | 1m 33s | | hadoop-yarn-server-web-proxy in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 530m 41s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus
|
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| Subsystem | Report/Notes |
|--
[GitHub] [hadoop] hadoop-yetus commented on pull request #3290: HADOOP-17843. Support IPV6 with IP for internal and external communication
hadoop-yetus commented on pull request #3290:
URL: https://github.com/apache/hadoop/pull/3290#issuecomment-915658156
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ HADOOP-17800 Compile Tests _ |
| +0 :ok: | mvndep | 11m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 57s | | HADOOP-17800 passed |
| +1 :green_heart: | compile | 23m 3s | | HADOOP-17800 passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 26s | | HADOOP-17800 passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 54s | | HADOOP-17800 passed |
| +1 :green_heart: | mvnsite | 2m 59s | | HADOOP-17800 passed |
| +1 :green_heart: | javadoc | 2m 28s | | HADOOP-17800 passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 57s | | HADOOP-17800 passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 0s | | HADOOP-17800 passed |
| +1 :green_heart: | shadedclient | 16m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 13s | | the patch passed |
| +1 :green_heart: | compile | 22m 8s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 8s | | the patch passed |
| +1 :green_heart: | compile | 19m 33s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 50s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3290/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 69 unchanged - 0 fixed = 70 total (was
69) |
| +1 :green_heart: | mvnsite | 3m 1s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 26s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 59s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 329m 36s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3290/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 0m 58s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3290/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
| hadoop-yarn-common in the patch failed. |
| +1 :green_heart: | unit | 1m 33s | | hadoop-yarn-server-web-proxy in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 530m 41s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus
|
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3290/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3290 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 6580a21872f0 4.15.0-147-generic #151-Ubuntu SMP Fri J
[GitHub] [hadoop] hadoop-yetus commented on pull request #2971: MAPREDUCE-7341. Intermediate Manifest Committer
hadoop-yetus commented on pull request #2971:
URL: https://github.com/apache/hadoop/pull/2971#issuecomment-915655232
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 23 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 53s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 13s | | trunk passed |
| +1 :green_heart: | compile | 28m 31s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 24m 18s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 16s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 9s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 36s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 36s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 18m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 43s | | the patch passed |
| +1 :green_heart: | compile | 28m 21s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| -1 :x: | javac | 28m 21s |
[/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/33/artifact/out/results-compile-javac-root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| root-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 generated 2 new + 1913 unchanged - 0
fixed = 1915 total (was 1913) |
| +1 :green_heart: | compile | 24m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | javac | 24m 19s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/33/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| root-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 generated 2 new + 1790
unchanged - 0 fixed = 1792 total (was 1790) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 21s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/33/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 29 new + 6 unchanged - 0 fixed = 35 total (was 6)
|
| +1 :green_heart: | mvnsite | 4m 20s | | the patch passed |
| +1 :green_heart: | xml | 0m 11s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 3m 8s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 37s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 34s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 18m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 31s | | hadoop-project in the patch
passed. |
| -1 :x: | unit | 19m 27s |
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2971/33/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 9m 13s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 2m 31s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not
generate ASF License warnings. |
| | | 267m 26s | | |
| Reason | Tests |
|---:|
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648267&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648267
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 08/Sep/21 23:12
Start Date: 08/Sep/21 23:12
Worklog Time Spent: 10m
Work Description: sunchao commented on a change in pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#discussion_r704836325
##
File path:
hadoop-client-modules/hadoop-client-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh
##
@@ -67,6 +67,8 @@ allowed_expr+="|^krb5_udp-template.conf$"
# Jetty uses this style sheet for directory listings. TODO ensure our
# internal use of jetty disallows directory listings and remove this.
allowed_expr+="|^jetty-dir.css$"
+# Snappy java is native library. We cannot relocate it to under
org/apache/hadoop.
Review comment:
What is `RegisterNatives`? is it something exists in snappy-java?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648267)
Time Spent: 8h 10m (was: 8h)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on a change in pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
sunchao commented on a change in pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#discussion_r704836325 ## File path: hadoop-client-modules/hadoop-client-check-invariants/src/test/resources/ensure-jars-have-correct-contents.sh ## @@ -67,6 +67,8 @@ allowed_expr+="|^krb5_udp-template.conf$" # Jetty uses this style sheet for directory listings. TODO ensure our # internal use of jetty disallows directory listings and remove this. allowed_expr+="|^jetty-dir.css$" +# Snappy java is native library. We cannot relocate it to under org/apache/hadoop. Review comment: What is `RegisterNatives`? is it something exists in snappy-java? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648266&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648266
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 08/Sep/21 23:11
Start Date: 08/Sep/21 23:11
Worklog Time Spent: 10m
Work Description: sunchao commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915633092
Yea I think it's OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648266)
Time Spent: 8h (was: 7h 50m)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
sunchao commented on pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915633092 Yea I think it's OK -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648259&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648259
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 08/Sep/21 22:55
Start Date: 08/Sep/21 22:55
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-914879385
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 15s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 8s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 5s | | trunk passed |
| +1 :green_heart: | compile | 0m 20s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 52s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 49s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-api no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 21s | |
branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 20s | |
branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output
file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 21s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 21s | |
branch/hadoop-client-modules/hadoop-client-integration-tests no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 17m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 13m 5s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 14s |
[/results-checkstyle-hadoop-client-modules.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/8/artifact/out/results-checkstyle-hadoop-client-modules.txt)
| hadoop-client-modules: The patch generated 15 new + 0 unchanged - 0 fixed =
15 total (was 0) |
| +1 :green_heart: | mvnsite | 1m 19s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 13s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 13s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-api has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 17s | |
hadoop-client-modules/hadoop-client-runtime has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
|
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-integration-tests has n
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648258&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648258
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 08/Sep/21 22:55
Start Date: 08/Sep/21 22:55
Worklog Time Spent: 10m
Work Description: hadoop-yetus removed a comment on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915080785
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648258)
Time Spent: 7h 40m (was: 7.5h)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
hadoop-yetus removed a comment on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-914879385
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 15s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 8s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 5s | | trunk passed |
| +1 :green_heart: | compile | 0m 20s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 52s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 49s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-api no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 21s | |
branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 20s | |
branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output
file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 21s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 21s | |
branch/hadoop-client-modules/hadoop-client-integration-tests no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 17m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 13m 5s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 14s |
[/results-checkstyle-hadoop-client-modules.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3385/8/artifact/out/results-checkstyle-hadoop-client-modules.txt)
| hadoop-client-modules: The patch generated 15 new + 0 unchanged - 0 fixed =
15 total (was 0) |
| +1 :green_heart: | mvnsite | 1m 19s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 13s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 13s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-api has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 17s | |
hadoop-client-modules/hadoop-client-runtime has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
|
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-integration-tests has no data from spotbugs
|
| +1 :green_heart: | shadedclient | 18m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-api in the patch
passed. |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-runtime in the
patch passed. |
| +1 :green_heart: | unit | 0m 15s | | hadoop-client-check-
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
hadoop-yetus removed a comment on pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915080785 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17804) Prometheus metrics only include the last set of labels
[ https://issues.apache.org/jira/browse/HADOOP-17804?focusedWorklogId=648256&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648256 ] ASF GitHub Bot logged work on HADOOP-17804: --- Author: ASF GitHub Bot Created on: 08/Sep/21 22:51 Start Date: 08/Sep/21 22:51 Worklog Time Spent: 10m Work Description: Kimahriman commented on pull request #3369: URL: https://github.com/apache/hadoop/pull/3369#issuecomment-915625266 Seems to be a race condition in the test itself, investigating... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 648256) Time Spent: 2.5h (was: 2h 20m) > Prometheus metrics only include the last set of labels > -- > > Key: HADOOP-17804 > URL: https://issues.apache.org/jira/browse/HADOOP-17804 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 3.3.1 >Reporter: Adam Binford >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > A prometheus endpoint was added in > https://issues.apache.org/jira/browse/HADOOP-16398, but the logic that puts > them into a map based on the "key" incorrectly hides any metrics with the > same key but different labels. The relevant code is here: > [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/PrometheusMetricsSink.java#L55|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/PrometheusMetricsSink.java#L55.] > The labels/tags need to be taken into account, as different tags mean > different metrics. For example, I came across this while trying to scrape > metrics for all the queues in our scheduler. Only the last queue is included > because all the metrics have the same "key" but a different "queue" label/tag. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Kimahriman commented on pull request #3369: HADOOP-17804. Expose prometheus metrics only after a flush and dedupe with tag values
Kimahriman commented on pull request #3369: URL: https://github.com/apache/hadoop/pull/3369#issuecomment-915625266 Seems to be a race condition in the test itself, investigating... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648254&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648254
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 08/Sep/21 22:45
Start Date: 08/Sep/21 22:45
Worklog Time Spent: 10m
Work Description: viirya edited a comment on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915622649
@dbtsai suggested that we exclude snappy-java artifacts from the shaded
client libraries. So we leverage maven to resolve difference between Hadoop's
snappy-java and downstream (e.g. Spark)'s snappy-java versions. Because for
snappy-java, its api should be stable, but there might be conflicts between
implementation details (e.g. bug fix), so sounds make sense. @sunchao WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648254)
Time Spent: 7.5h (was: 7h 20m)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya edited a comment on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
viirya edited a comment on pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915622649 @dbtsai suggested that we exclude snappy-java artifacts from the shaded client libraries. So we leverage maven to resolve difference between Hadoop's snappy-java and downstream (e.g. Spark)'s snappy-java versions. Because for snappy-java, its api should be stable, but there might be conflicts between implementation details (e.g. bug fix), so sounds make sense. @sunchao WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648253&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648253
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 08/Sep/21 22:44
Start Date: 08/Sep/21 22:44
Worklog Time Spent: 10m
Work Description: viirya commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915622649
@dbtsai suggested that we exclude snappy-java artifacts from the shaded
client libraries. So we leverage maven to resolve difference between Hadoop's
snappy-java and downstream (e.g. Spark)'s snappy-java versions. Sounds make
sense. @sunchao WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648253)
Time Spent: 7h 20m (was: 7h 10m)
> lz4-java and snappy-java should be excluded from relocation in shaded Hadoop
> libraries
> --
>
> Key: HADOOP-17891
> URL: https://issues.apache.org/jira/browse/HADOOP-17891
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> lz4-java is a provided dependency. So in the shaded Hadoop libraries, e.g.
> hadoop-client-api, if we don't exclude lz4 dependency, the downstream will
> still see the exception even they include lz4 dependency.
> {code:java}
> [info] Cause: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.net.jpountz.lz4.LZ4Factory
> [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> [info] at
> org.apache.hadoop.io.compress.lz4.Lz4Compressor.(Lz4Compressor.java:66)
> [info] at
> org.apache.hadoop.io.compress.Lz4Codec.createCompressor(Lz4Codec.java:119)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:152)
> [info] at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
> {code}
> Currently snappy-java is included and relocated in Hadoop shaded client
> libraries. But as it includes native methods, it should not be relocated too
> due to JNI method resolution. The downstream will see the exception:
> {code}
> [info] Cause: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
> Method)
>
> [info] at
> org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
>
>
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
> [info] at
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] viirya commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
viirya commented on pull request #3385: URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915622649 @dbtsai suggested that we exclude snappy-java artifacts from the shaded client libraries. So we leverage maven to resolve difference between Hadoop's snappy-java and downstream (e.g. Spark)'s snappy-java versions. Sounds make sense. @sunchao WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] LeonGao91 commented on pull request #3386: HDFS-16213. De-flake TestFsDatasetImpl#testDnRestartWithHardLink
LeonGao91 commented on pull request #3386:
URL: https://github.com/apache/hadoop/pull/3386#issuecomment-915598330
Thanks @virajjasani for reporting this issue!
Seems like this happens consistently when running the same test multiple
times, but doesn't fail when running it the first time (happy case), like you
mentioned, this can reproduce it consistently:
@Test
public void t1() throws Exception {
testDnRestartWithHardLink();
testDnRestartWithHardLink();
}
Based on this I am not sure if the root cause is the rare race condition you
mentioned. I suspect it is due to the cache behavior when rerun the same code
the replica loading changed a little bit.
I can spend some more time investigating in the next few days. Let's try to
fix this in the unit test itself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17891) lz4-java and snappy-java should be excluded from relocation in shaded Hadoop libraries
[
https://issues.apache.org/jira/browse/HADOOP-17891?focusedWorklogId=648216&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648216
]
ASF GitHub Bot logged work on HADOOP-17891:
---
Author: ASF GitHub Bot
Created on: 08/Sep/21 21:04
Start Date: 08/Sep/21 21:04
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915569374
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 30s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 0s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 57s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 28s | |
branch/hadoop-client-modules/hadoop-client-api no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output
file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-integration-tests no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 26m 37s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 10m 1s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 14s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 17s | |
hadoop-client-modules/hadoop-client-api has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-runtime has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
|
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-integration-tests has no data from spotbugs
|
| +1 :green_heart: | shadedclient | 27m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-api in the pat
[GitHub] [hadoop] hadoop-yetus commented on pull request #3385: HADOOP-17891. Exclude snappy-java and lz4-java from relocation in shaded hadoop client libraries
hadoop-yetus commented on pull request #3385:
URL: https://github.com/apache/hadoop/pull/3385#issuecomment-915569374
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 30s | | trunk passed |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 0s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 57s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 28s | |
branch/hadoop-client-modules/hadoop-client-api no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 25s | |
branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output
file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 24s | |
branch/hadoop-client-modules/hadoop-client-integration-tests no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 26m 37s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 10m 1s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | xml | 0m 5s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 14s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 17s | |
hadoop-client-modules/hadoop-client-api has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-runtime has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
|
| +0 :ok: | spotbugs | 0m 15s | |
hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs |
| +0 :ok: | spotbugs | 0m 16s | |
hadoop-client-modules/hadoop-client-integration-tests has no data from spotbugs
|
| +1 :green_heart: | shadedclient | 27m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-api in the patch
passed. |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-runtime in the
patch passed. |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-check-invariants
in the patch passed. |
| +1 :green_heart: | unit | 0m 19s | | hadoop-client-minicluster in
the patch passed. |
| +1 :green_heart: | unit | 0m 17s | | hadoop-client-integration-tests
in the patch passed. |
| +1 :green_
[jira] [Updated] (HADOOP-17902) Fix Hadoop build on Debian 10
[
https://issues.apache.org/jira/browse/HADOOP-17902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-17902:
Labels: pull-request-available (was: )
> Fix Hadoop build on Debian 10
> -
>
> Key: HADOOP-17902
> URL: https://issues.apache.org/jira/browse/HADOOP-17902
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We're using *Debian testing* as one of the package sources to get the latest
> packages. It seems to be broken at the moment. The CI fails to create the
> build environment for the Debian 10 platform -
> {code}
> [2021-09-08T00:21:11.596Z] #13 [ 8/14] RUN apt-get -q update && apt-get
> -q install -y --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*
> ...
> [2021-09-08T00:21:22.744Z] #13 11.28 Preparing to unpack
> .../libc6_2.31-17_amd64.deb ...
> [2021-09-08T00:21:23.260Z] #13 11.46 Checking for services that may need to
> be restarted...
> [2021-09-08T00:21:23.260Z] #13 11.48 Checking init scripts...
> [2021-09-08T00:21:23.260Z] #13 11.50 Unpacking libc6:amd64 (2.31-17) over
> (2.28-10) ...
> [2021-09-08T00:21:26.290Z] #13 14.38 Setting up libc6:amd64 (2.31-17) ...
> [2021-09-08T00:21:26.290Z] #13 14.42 /usr/bin/perl: error while loading
> shared libraries: libcrypt.so.1: cannot open shared object file: No such file
> or directory
> [2021-09-08T00:21:26.290Z] #13 14.42 dpkg: error processing package
> libc6:amd64 (--configure):
> [2021-09-08T00:21:26.290Z] #13 14.42 installed libc6:amd64 package
> post-installation script subprocess returned error exit status 127
> [2021-09-08T00:21:26.291Z] #13 14.43 Errors were encountered while processing:
> [2021-09-08T00:21:26.291Z] #13 14.43 libc6:amd64
> [2021-09-08T00:21:26.291Z] #13 14.46 E: Sub-process /usr/bin/dpkg returned an
> error code (1)
> [2021-09-08T00:21:27.867Z] #13 ERROR: executor failed running [/bin/bash -o
> pipefail -c apt-get -q update && apt-get -q install -y
> --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*]: exit code: 100
> [2021-09-08T00:21:27.867Z] --
> [2021-09-08T00:21:27.867Z] > [ 8/14] RUN apt-get -q update && apt-get -q
> install -y --no-install-recommends python3 && apt-get -q install -y
> --no-install-recommends $(pkg-resolver/resolve.py debian:10) && apt-get
> clean && rm -rf /var/lib/apt/lists/*:
> [2021-09-08T00:21:27.867Z] --
> [2021-09-08T00:21:27.867Z] executor failed running [/bin/bash -o pipefail -c
> apt-get -q update && apt-get -q install -y --no-install-recommends
> python3 && apt-get -q install -y --no-install-recommends
> $(pkg-resolver/resolve.py debian:10) && apt-get clean && rm -rf
> /var/lib/apt/lists/*]: exit code: 100
> [2021-09-08T00:21:27.867Z] ERROR: Docker failed to build
> yetus/hadoop:ef5dbc7283a.
> [2021-09-08T00:21:27.867Z]
> {code}
> The above log lines are copied from -
> https://ci-hadoop.apache.org/blue/organizations/jenkins/hadoop-multibranch/detail/PR-3388/3/pipeline
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17902) Fix Hadoop build on Debian 10
[
https://issues.apache.org/jira/browse/HADOOP-17902?focusedWorklogId=648214&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648214
]
ASF GitHub Bot logged work on HADOOP-17902:
---
Author: ASF GitHub Bot
Created on: 08/Sep/21 21:01
Start Date: 08/Sep/21 21:01
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#issuecomment-915567561
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 43m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 1s | | trunk passed |
| +1 :green_heart: | compile | 2m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 53m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 2m 25s | | the patch passed |
| +1 :green_heart: | cc | 2m 25s | | the patch passed |
| +1 :green_heart: | golang | 2m 25s | | the patch passed |
| +1 :green_heart: | javac | 2m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 62m 57s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt)
| hadoop-hdfs-native-client in the patch failed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 181m 6s | | |
| Reason | Tests |
|---:|:--|
| Failed CTEST tests | test_libhdfs_threaded_hdfspp_test_shim_static |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3408 |
| Optional Tests | dupname asflicense codespell hadolint shellcheck
shelldocs compile cc mvnsite javac unit golang |
| uname | Linux 2a8810d8c975 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13
01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a15b580c2b523f0950ce5d703ebd14b8929094a3 |
| Default Java | Red Hat, Inc.-1.8.0_302-b08 |
| CTEST |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/1/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/1/testReport/ |
| Max. process+thread count | 616 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/1/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 648214)
Remaining Estimate: 0h
Time Spent: 10m
> Fix Hadoop build on Debian 10
>
[GitHub] [hadoop] hadoop-yetus commented on pull request #3408: HADOOP-17902. Fix Hadoop build on Debian 10
hadoop-yetus commented on pull request #3408:
URL: https://github.com/apache/hadoop/pull/3408#issuecomment-915567561
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 43m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | hadolint | 0m 0s | | hadolint was not available. |
| +0 :ok: | shellcheck | 0m 0s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 1s | | trunk passed |
| +1 :green_heart: | compile | 2m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 53m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 2m 25s | | the patch passed |
| +1 :green_heart: | cc | 2m 25s | | the patch passed |
| +1 :green_heart: | golang | 2m 25s | | the patch passed |
| +1 :green_heart: | javac | 2m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 62m 57s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt)
| hadoop-hdfs-native-client in the patch failed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 181m 6s | | |
| Reason | Tests |
|---:|:--|
| Failed CTEST tests | test_libhdfs_threaded_hdfspp_test_shim_static |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3408 |
| Optional Tests | dupname asflicense codespell hadolint shellcheck
shelldocs compile cc mvnsite javac unit golang |
| uname | Linux 2a8810d8c975 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13
01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a15b580c2b523f0950ce5d703ebd14b8929094a3 |
| Default Java | Red Hat, Inc.-1.8.0_302-b08 |
| CTEST |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/1/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/1/testReport/ |
| Max. process+thread count | 616 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3408/1/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3407: YARN-10910. AbstractCSQueue#setupQueueConfigs: Separate validation lo…
hadoop-yetus commented on pull request #3407:
URL: https://github.com/apache/hadoop/pull/3407#issuecomment-915553997
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 10s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 29s | | trunk passed |
| +1 :green_heart: | compile | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | compile | 1m 0s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 0s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 42s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 0s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 42s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 11s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 101m 4s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 190m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3407/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 257e56c4b943 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 198dfefdd0d700f3fcfebdb31db803ebb1e60071 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3407/1/testReport/ |
| Max. process+thread count | 901 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3407/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
[GitHub] [hadoop] hadoop-yetus commented on pull request #2281: HDFS-15516.Add info for create flags in NameNode audit logs.
hadoop-yetus commented on pull request #2281: URL: https://github.com/apache/hadoop/pull/2281#issuecomment-915553180 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 26s | | https://github.com/apache/hadoop/pull/2281 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/2281 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2281/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3407: YARN-10910. AbstractCSQueue#setupQueueConfigs: Separate validation lo…
hadoop-yetus commented on pull request #3407:
URL: https://github.com/apache/hadoop/pull/3407#issuecomment-915551542
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 5s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 38s | | trunk passed |
| +1 :green_heart: | compile | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 59s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 0m 47s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 47s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 38s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 99m 47s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 183m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3407/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3407 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux a8901dff2aaf 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 198dfefdd0d700f3fcfebdb31db803ebb1e60071 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3407/2/testReport/ |
| Max. process+thread count | 912 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3407/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
[jira] [Work logged] (HADOOP-17195) Intermittent OutOfMemory error while performing hdfs CopyFromLocal to abfs
[
https://issues.apache.org/jira/browse/HADOOP-17195?focusedWorklogId=648192&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-648192
]
ASF GitHub Bot logged work on HADOOP-17195:
---
Author: ASF GitHub Bot
Created on: 08/Sep/21 19:52
Start Date: 08/Sep/21 19:52
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3406:
URL: https://github.com/apache/hadoop/pull/3406#issuecomment-915524514
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 11m 21s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 57s | | trunk passed |
| +1 :green_heart: | compile | 23m 2s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 4s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 10s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 36s | | the patch passed |
| +1 :green_heart: | compile | 23m 52s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 52s | | the patch passed |
| +1 :green_heart: | compile | 20m 40s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 20m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 57s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3406/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 5 new + 74 unchanged - 0 fixed = 79 total (was
74) |
| +1 :green_heart: | mvnsite | 2m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 40s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 17s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 2m 45s |
[/new-spotbugs-hadoop-common-project_hadoop-common.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3406/1/artifact/out/new-spotbugs-hadoop-common-project_hadoop-common.html)
| hadoop-common-project/hadoop-common generated 2 new + 0 unchanged - 0 fixed
= 2 total (was 0) |
| +1 :green_heart: | shadedclient | 17m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 29s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 11s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 211m 25s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-common-project/hadoop-common |
| | Exceptional return value of java.io.File.delete() ignored in
org.apache.hadoop.fs.store.DataBlocks$BlockUploadData.close() At
DataBlocks.java:ignored in
org.apache.hadoop.fs.store.DataBlocks$BlockUploadData.close() At
DataBlocks.java:[line 212] |
| | Write to static field
org.apache.hadoop.fs.store.DataBlocks$DiskBlockFactory.directoryAllocator from
instance method new
org.apache.hadoop.fs.store.DataBlocks$DiskBlockFactory(String, Configuration)
At DataBlocks.java:from instance method new
org.apache.hadoop.fs.store.DataBlocks$DiskBlockFactory(String, Configuration)
At DataBlocks.java:[line
[GitHub] [hadoop] hadoop-yetus commented on pull request #3406: HADOOP-17195. OutOfMemory error while performing hdfs CopyFromLocal to ABFS
hadoop-yetus commented on pull request #3406:
URL: https://github.com/apache/hadoop/pull/3406#issuecomment-915524514
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 11m 21s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 57s | | trunk passed |
| +1 :green_heart: | compile | 23m 2s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 4s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 4m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 10s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 36s | | the patch passed |
| +1 :green_heart: | compile | 23m 52s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 23m 52s | | the patch passed |
| +1 :green_heart: | compile | 20m 40s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 20m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 57s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3406/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 5 new + 74 unchanged - 0 fixed = 79 total (was
74) |
| +1 :green_heart: | mvnsite | 2m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 40s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 17s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| -1 :x: | spotbugs | 2m 45s |
[/new-spotbugs-hadoop-common-project_hadoop-common.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3406/1/artifact/out/new-spotbugs-hadoop-common-project_hadoop-common.html)
| hadoop-common-project/hadoop-common generated 2 new + 0 unchanged - 0 fixed
= 2 total (was 0) |
| +1 :green_heart: | shadedclient | 17m 13s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 29s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 11s | | hadoop-azure in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 211m 25s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-common-project/hadoop-common |
| | Exceptional return value of java.io.File.delete() ignored in
org.apache.hadoop.fs.store.DataBlocks$BlockUploadData.close() At
DataBlocks.java:ignored in
org.apache.hadoop.fs.store.DataBlocks$BlockUploadData.close() At
DataBlocks.java:[line 212] |
| | Write to static field
org.apache.hadoop.fs.store.DataBlocks$DiskBlockFactory.directoryAllocator from
instance method new
org.apache.hadoop.fs.store.DataBlocks$DiskBlockFactory(String, Configuration)
At DataBlocks.java:from instance method new
org.apache.hadoop.fs.store.DataBlocks$DiskBlockFactory(String, Configuration)
At DataBlocks.java:[line 884] |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3406/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3406 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | L
[GitHub] [hadoop] hadoop-yetus commented on pull request #3386: HDFS-16213. De-flake TestFsDatasetImpl#testDnRestartWithHardLink
hadoop-yetus commented on pull request #3386:
URL: https://github.com/apache/hadoop/pull/3386#issuecomment-915516780
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 42s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 25s | | trunk passed |
| +1 :green_heart: | compile | 2m 1s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 49s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 4s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 35s | | the patch passed |
| +1 :green_heart: | compile | 1m 33s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 58s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 240 unchanged - 3
fixed = 240 total (was 243) |
| +1 :green_heart: | mvnsite | 1m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 46s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 391m 45s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3386/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 511m 12s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestBlockTokenWrappingQOP |
| | hadoop.hdfs.TestReconstructStripedFileWithValidator |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy |
| | hadoop.hdfs.tools.TestECAdmin |
| | hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| | hadoop.hdfs.TestWriteConfigurationToDFS |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3386/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3386 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 889f73ecc5d5 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 763f43cb14cf3385538e37f40cecc763c7450fa5 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3386/7/testRep
