[GitHub] [hadoop] hadoop-yetus commented on issue #1673: HDFS-14638. [Dynamometer] Fix scripts to refer to current build structure.
hadoop-yetus commented on issue #1673: HDFS-14638. [Dynamometer] Fix scripts to refer to current build structure. URL: https://github.com/apache/hadoop/pull/1673#issuecomment-545735831 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 77 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | 0 | shelldocs | 0 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 34 | Maven dependency ordering for branch | | +1 | mvninstall | 1234 | trunk passed | | +1 | mvnsite | 80 | trunk passed | | +1 | shadedclient | 808 | branch has no errors when building and testing our client artifacts. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 13 | Maven dependency ordering for patch | | +1 | mvninstall | 72 | the patch passed | | +1 | mvnsite | 61 | the patch passed | | +1 | shellcheck | 1 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 896 | patch has no errors when building and testing our client artifacts. | ||| _ Other Tests _ | | +1 | unit | 21 | hadoop-dynamometer-workload in the patch passed. | | +1 | unit | 22 | hadoop-dynamometer-infra in the patch passed. | | +1 | unit | 21 | hadoop-dynamometer-blockgen in the patch passed. | | +1 | asflicense | 32 | The patch does not generate ASF License warnings. | | | | 3497 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1673/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1673 | | Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs | | uname | Linux 22665cb4a1c6 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / fd84ca5 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1673/1/testReport/ | | Max. process+thread count | 342 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-workload hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen U: hadoop-tools/hadoop-dynamometer | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1673/1/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on issue #1673: HDFS-14638. [Dynamometer] Fix scripts to refer to current build structure.
jojochuang commented on issue #1673: HDFS-14638. [Dynamometer] Fix scripts to refer to current build structure. URL: https://github.com/apache/hadoop/pull/1673#issuecomment-545732915 @smengcl would you like to review this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tasanuma opened a new pull request #1673: HDFS-14638. [Dynamometer] Fix scripts to refer to current build structure.
tasanuma opened a new pull request #1673: HDFS-14638. [Dynamometer] Fix scripts to refer to current build structure. URL: https://github.com/apache/hadoop/pull/1673 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1670: HDFS-14925. Rename operation should check nest snapshot
hadoop-yetus commented on issue #1670: HDFS-14925. Rename operation should check nest snapshot URL: https://github.com/apache/hadoop/pull/1670#issuecomment-545708834 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 2112 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1112 | trunk passed | | +1 | compile | 66 | trunk passed | | +1 | checkstyle | 48 | trunk passed | | +1 | mvnsite | 75 | trunk passed | | +1 | shadedclient | 856 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 85 | trunk passed | | 0 | spotbugs | 168 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 165 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 65 | the patch passed | | +1 | compile | 62 | the patch passed | | +1 | javac | 62 | the patch passed | | -0 | checkstyle | 40 | hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 28 unchanged - 0 fixed = 30 total (was 28) | | +1 | mvnsite | 66 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 794 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 77 | the patch passed | | +1 | findbugs | 168 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 8200 | hadoop-hdfs in the patch failed. | | +1 | asflicense | 42 | The patch does not generate ASF License warnings. | | | | 14114 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.namenode.TestRedudantBlocks | | | hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing | | | hadoop.hdfs.server.namenode.snapshot.TestRandomOpsWithSnapshots | | | hadoop.hdfs.server.namenode.TestFsck | | | hadoop.hdfs.server.namenode.ha.TestHAAppend | | | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots | | | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1670/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1670 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux a017f944f4fe 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 1d5d7d0 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1670/2/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1670/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1670/2/testReport/ | | Max. process+thread count | 3910 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1670/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1672: HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException
hadoop-yetus commented on issue #1672: HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException URL: https://github.com/apache/hadoop/pull/1672#issuecomment-545705717 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 159 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1642 | trunk passed | | +1 | compile | 94 | trunk passed | | +1 | checkstyle | 61 | trunk passed | | +1 | mvnsite | 93 | trunk passed | | +1 | shadedclient | 1109 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 89 | trunk passed | | 0 | spotbugs | 217 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 216 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 87 | the patch passed | | +1 | compile | 72 | the patch passed | | +1 | javac | 72 | the patch passed | | +1 | checkstyle | 48 | the patch passed | | +1 | mvnsite | 87 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 1038 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 91 | the patch passed | | +1 | findbugs | 220 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 6603 | hadoop-hdfs in the patch failed. | | +1 | asflicense | 35 | The patch does not generate ASF License warnings. | | | | 11813 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeMXBean | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1672/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1672 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 38856c08363d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 1d5d7d0 | | Default Java | 1.8.0_222 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1672/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1672/1/testReport/ | | Max. process+thread count | 2979 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1672/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhjwpku commented on issue #1670: HDFS-14925. Rename operation should check nest snapshot
zhjwpku commented on issue #1670: HDFS-14925. Rename operation should check nest snapshot URL: https://github.com/apache/hadoop/pull/1670#issuecomment-545695573 This patch misses considering dir to dir overwrite scenario, will modify and push later, thx. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1671: HADOOP-16665. Filesystems to be closed if they failed during initialize().
hadoop-yetus commented on issue #1671: HADOOP-16665. Filesystems to be closed if they failed during initialize(). URL: https://github.com/apache/hadoop/pull/1671#issuecomment-545666451 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 2196 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 105 | Maven dependency ordering for branch | | +1 | mvninstall | 2445 | trunk passed | | +1 | compile | 2266 | trunk passed | | +1 | checkstyle | 180 | trunk passed | | +1 | mvnsite | 134 | trunk passed | | +1 | shadedclient | 1453 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 370 | trunk passed | | 0 | spotbugs | 235 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 598 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 63 | Maven dependency ordering for patch | | +1 | mvninstall | 216 | the patch passed | | +1 | compile | 2218 | the patch passed | | +1 | javac | 2218 | the patch passed | | -0 | checkstyle | 168 | root: The patch generated 1 new + 97 unchanged - 0 fixed = 98 total (was 97) | | +1 | mvnsite | 122 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 840 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 120 | the patch passed | | +1 | findbugs | 206 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 540 | hadoop-common in the patch failed. | | +1 | unit | 87 | hadoop-aws in the patch passed. | | +1 | asflicense | 46 | The patch does not generate ASF License warnings. | | | | 14531 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.security.TestRaceWhenRelogin | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1671/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1671 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 55c616192b71 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 1d5d7d0 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1671/1/artifact/out/diff-checkstyle-root.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1671/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1671/1/testReport/ | | Max. process+thread count | 1411 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1671/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang opened a new pull request #1672: HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException
jojochuang opened a new pull request #1672: HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException URL: https://github.com/apache/hadoop/pull/1672 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] dineshchitlangia commented on a change in pull request #1670: HDFS-14925. Rename operation should check nest snapshot
dineshchitlangia commented on a change in pull request #1670: HDFS-14925.
Rename operation should check nest snapshot
URL: https://github.com/apache/hadoop/pull/1670#discussion_r338241039
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
##
@@ -400,6 +408,13 @@ static RenameResult unprotectedRenameTo(FSDirectory fsd,
throw new ParentNotDirectoryException(error);
}
+if (snapshottableDirs.size() > 0) {
+ if (dstParent.asDirectory().isDescendantOfSnapshotRoot()) {
+throw new SnapshotException("Failed to rename " + src + " to " + dst +
+" due to nested snapshottable directory");
+ }
+}
+
Review comment:
This code snippet is written twice in same class, in different methods. Can
we instead carve it into a method to avoid code duplication?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1670: HDFS-14925. Rename operation should check nest snapshot
hadoop-yetus commented on issue #1670: HDFS-14925. Rename operation should check nest snapshot URL: https://github.com/apache/hadoop/pull/1670#issuecomment-545602150 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 2075 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1271 | trunk passed | | +1 | compile | 65 | trunk passed | | +1 | checkstyle | 47 | trunk passed | | +1 | mvnsite | 71 | trunk passed | | +1 | shadedclient | 957 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 82 | trunk passed | | 0 | spotbugs | 180 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 177 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 65 | the patch passed | | +1 | compile | 61 | the patch passed | | +1 | javac | 61 | the patch passed | | -0 | checkstyle | 42 | hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 28 unchanged - 0 fixed = 30 total (was 28) | | +1 | mvnsite | 66 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 886 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 75 | the patch passed | | +1 | findbugs | 180 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 6257 | hadoop-hdfs in the patch failed. | | +1 | asflicense | 35 | The patch does not generate ASF License warnings. | | | | 12478 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing | | | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots | | | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency | | | hadoop.hdfs.TestReconstructStripedFile | | | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1670/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1670 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux f5150bcc6203 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 1d5d7d0 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1670/1/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1670/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1670/1/testReport/ | | Max. process+thread count | 2829 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1670/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1671: HADOOP-16665. Filesystems to be closed if they failed during initialize().
steveloughran commented on issue #1671: HADOOP-16665. Filesystems to be closed if they failed during initialize(). URL: https://github.com/apache/hadoop/pull/1671#issuecomment-545585572 regression testing in progress. I haven't written any tests to explicitly explore the new codepaths. I will probably add a new common test which has a mock file system that fails during init and whose close() updates a static counter. Adding a test for S3A's explicit cleanup is a bit trickier. I could probably add an ITest which tried to create an FS with invalid settings -encryption ones get checked first- and then close() that after. This will at least show that the cleanup code is re-entrant. Trying to force a failure at a specific point after some closable items have been created (instrumentation, thread pools etc) and verifying that these are null afterwards is going to be very brittle. Avoid. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran opened a new pull request #1671: HADOOP-16665. Filesystems to be closed if they failed during initialize().
steveloughran opened a new pull request #1671: HADOOP-16665. Filesystems to be closed if they failed during initialize(). URL: https://github.com/apache/hadoop/pull/1671 Patches FileSystem to do this, and for S3A have it stop services during init exception handling. Also reviewed the mockFS subclass to make sure it gets close() right and remove all uses of the obsolete S3AUtils.closeAll with IOUtils.cleanupWithLogger Change-Id: I0b9fd1aafc902a151a8cacb614058e00d475cd45 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16665) Filesystems to be closed if they failed during initialize()
[ https://issues.apache.org/jira/browse/HADOOP-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16958151#comment-16958151 ] Steve Loughran commented on HADOOP-16665: - I'm implementing robust cleanup in s3afs init in this patch too > Filesystems to be closed if they failed during initialize() > --- > > Key: HADOOP-16665 > URL: https://issues.apache.org/jira/browse/HADOOP-16665 > Project: Hadoop Common > Issue Type: Bug > Components: fs, fs/s3 >Affects Versions: 3.2.1, 3.1.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > We seen this with S3A but it is likely to apply to the other file systems too: > If the file system raises an exception during initialize() then it is not > close()'d. Any resources threads etc created during initialisation may be > leaked. > I'm going to fix S3A to clean up internally, but the FileSystem get() and > newInstance() methods should also catch and close if an IOE or RTE is raised. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16665) Filesystems to be closed if they failed during initialize()
[ https://issues.apache.org/jira/browse/HADOOP-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-16665: --- Assignee: Steve Loughran > Filesystems to be closed if they failed during initialize() > --- > > Key: HADOOP-16665 > URL: https://issues.apache.org/jira/browse/HADOOP-16665 > Project: Hadoop Common > Issue Type: Bug > Components: fs, fs/s3 >Affects Versions: 3.2.1, 3.1.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > We seen this with S3A but it is likely to apply to the other file systems too: > If the file system raises an exception during initialize() then it is not > close()'d. Any resources threads etc created during initialisation may be > leaked. > I'm going to fix S3A to clean up internally, but the FileSystem get() and > newInstance() methods should also catch and close if an IOE or RTE is raised. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] avijayanhwx commented on a change in pull request #1667: HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuff…
avijayanhwx commented on a change in pull request #1667: HDFS-14308.
DFSStripedInputStream curStripeBuf is not freed by unbuff…
URL: https://github.com/apache/hadoop/pull/1667#discussion_r338210796
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
##
@@ -554,4 +555,19 @@ public synchronized void releaseBuffer(ByteBuffer buffer)
{
throw new UnsupportedOperationException(
"Not support enhanced byte buffer access.");
}
+
+ @Override
+ public synchronized void unbuffer() {
+closeCurrentBlockReaders();
+if (curStripeBuf != null) {
+ curStripeBuf.clear();
Review comment:
Thanks for the changes. I am trying to understand why the parityBuf clear
step was removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] avijayanhwx commented on a change in pull request #1667: HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuff…
avijayanhwx commented on a change in pull request #1667: HDFS-14308.
DFSStripedInputStream curStripeBuf is not freed by unbuff…
URL: https://github.com/apache/hadoop/pull/1667#discussion_r338210796
##
File path:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
##
@@ -554,4 +555,19 @@ public synchronized void releaseBuffer(ByteBuffer buffer)
{
throw new UnsupportedOperationException(
"Not support enhanced byte buffer access.");
}
+
+ @Override
+ public synchronized void unbuffer() {
+closeCurrentBlockReaders();
+if (curStripeBuf != null) {
+ curStripeBuf.clear();
Review comment:
Thanks for the changes!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] cxorm commented on issue #1640: HADOOP-16637. Fix findbugs warnings in hadoop-cos.
cxorm commented on issue #1640: HADOOP-16637. Fix findbugs warnings in hadoop-cos. URL: https://github.com/apache/hadoop/pull/1640#issuecomment-545553004 > Most of the changes look good; that input stream cleanup looks like it will break the code as a closed stream will always get passed along to the next method. The findbugs file probably just needs changing. > > Which endpoint did you run the tests against? We just ran the findbugs for the file. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on issue #1670: HDFS-14925. Rename operation should check nest snapshot
jojochuang commented on issue #1670: HDFS-14925. Rename operation should check nest snapshot URL: https://github.com/apache/hadoop/pull/1670#issuecomment-545538271 @bshashikant could you help review this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1668: HADOOP-16645. S3A Delegation Token extension point to use StoreContext.
steveloughran commented on issue #1668: HADOOP-16645. S3A Delegation Token extension point to use StoreContext. URL: https://github.com/apache/hadoop/pull/1668#issuecomment-545531745 Happy to take the patch. I have my own PR to clean up the binding; https://github.com/apache/hadoop/pull/1668 As with yours, not backwards compatible. If you want to review that too, we could get them both into trunk, and then coordinate the backport This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1640: HADOOP-16637. Fix findbugs warnings in hadoop-cos.
steveloughran commented on issue #1640: HADOOP-16637. Fix findbugs warnings in hadoop-cos. URL: https://github.com/apache/hadoop/pull/1640#issuecomment-545529030 Most of the changes look good; that input stream cleanup looks like it will break the code as a closed stream will always get passed along to the next method. The findbugs file probably just needs changing. Which endpoint did you run the tests against? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1661: HADOOP-16484. S3A to warn or fail if S3Guard is disabled
hadoop-yetus commented on issue #1661: HADOOP-16484. S3A to warn or fail if S3Guard is disabled URL: https://github.com/apache/hadoop/pull/1661#issuecomment-545527968 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 36 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 26 | Maven dependency ordering for branch | | +1 | mvninstall | 1068 | trunk passed | | +1 | compile | 1038 | trunk passed | | +1 | checkstyle | 158 | trunk passed | | +1 | mvnsite | 137 | trunk passed | | +1 | shadedclient | 1114 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 133 | trunk passed | | 0 | spotbugs | 71 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 194 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for patch | | +1 | mvninstall | 82 | the patch passed | | +1 | compile | 982 | the patch passed | | +1 | javac | 982 | the patch passed | | -0 | checkstyle | 155 | root: The patch generated 2 new + 14 unchanged - 0 fixed = 16 total (was 14) | | +1 | mvnsite | 134 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 753 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 137 | the patch passed | | +1 | findbugs | 208 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 550 | hadoop-common in the patch passed. | | +1 | unit | 96 | hadoop-aws in the patch passed. | | +1 | asflicense | 55 | The patch does not generate ASF License warnings. | | | | 7093 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1661/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1661 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | uname | Linux 565af98172aa 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / a901405 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1661/3/artifact/out/diff-checkstyle-root.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1661/3/testReport/ | | Max. process+thread count | 1350 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1661/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1640: HADOOP-16637. Fix findbugs warnings in hadoop-cos.
steveloughran commented on a change in pull request #1640: HADOOP-16637. Fix
findbugs warnings in hadoop-cos.
URL: https://github.com/apache/hadoop/pull/1640#discussion_r338152515
##
File path:
hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNativeFileSystemStore.java
##
@@ -249,7 +250,15 @@ public void storeEmptyFile(String key) throws IOException
{
public PartETag uploadPart(File file, String key, String uploadId,
int partNum) throws IOException {
-InputStream inputStream = new FileInputStream(file);
+InputStream inputStream = null;
Review comment:
looks broken, as L262 will always get a closed stream
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1614: HADOOP-16615. Add password check for credential provider
steveloughran commented on issue #1614: HADOOP-16615. Add password check for
credential provider
URL: https://github.com/apache/hadoop/pull/1614#issuecomment-545526777
I like all tests in our code to provide enough diagnostics on failure that
we can work out what went wrong purely from the Jenkins logs.
AssertJ is really good here, which is why we are adopting it in new code.
For your tests, I'm going to propose a new assertion which can be used to
validate the output of the new command. If the condition is not met,
The exception raised will include all of the output.
```
private void assertOutputContains(String expected) {
Assertions.assertThat(outContent.toString())
.contains(expected));
}
```
Yetus probably isn't retesting the code, as it thinks the last change was a
documentation only change. If you change the test case things may be different.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1661: HADOOP-16484. S3A to warn or fail if S3Guard is disabled
steveloughran commented on issue #1661: HADOOP-16484. S3A to warn or fail if S3Guard is disabled URL: https://github.com/apache/hadoop/pull/1661#issuecomment-545518267 Thanks for the changes; looks good. One issue: is "org.apache.hadoop.fs.s3a.s3guard.disabled_warn_level" the right property name? I would rather see "." entries rather than underscores. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhjwpku commented on issue #1670: HDFS-14925. Rename operation should check nest snapshot
zhjwpku commented on issue #1670: HDFS-14925. Rename operation should check nest snapshot URL: https://github.com/apache/hadoop/pull/1670#issuecomment-545517326 @manojpec, please help to review this patch :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on issue #1661: HADOOP-16484. S3A to warn or fail if S3Guard is disabled
hadoop-yetus removed a comment on issue #1661: HADOOP-16484. S3A to warn or fail if S3Guard is disabled URL: https://github.com/apache/hadoop/pull/1661#issuecomment-543251581 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 2103 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 71 | Maven dependency ordering for branch | | +1 | mvninstall | 1089 | trunk passed | | +1 | compile | 1019 | trunk passed | | +1 | checkstyle | 161 | trunk passed | | +1 | mvnsite | 138 | trunk passed | | +1 | shadedclient | 1096 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 134 | trunk passed | | 0 | spotbugs | 72 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 196 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for patch | | +1 | mvninstall | 83 | the patch passed | | +1 | compile | 970 | the patch passed | | +1 | javac | 970 | the patch passed | | -0 | checkstyle | 159 | root: The patch generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14) | | +1 | mvnsite | 135 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 1 | The patch has no ill-formed XML file. | | +1 | shadedclient | 743 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 133 | the patch passed | | +1 | findbugs | 208 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 561 | hadoop-common in the patch failed. | | +1 | unit | 98 | hadoop-aws in the patch passed. | | +1 | asflicense | 56 | The patch does not generate ASF License warnings. | | | | 9207 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.conf.TestCommonConfigurationFields | | | hadoop.fs.shell.TestCopyFromLocal | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.3 Server=19.03.3 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1661/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1661 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | uname | Linux a99c1307be78 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 3990ffa | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1661/2/artifact/out/diff-checkstyle-root.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1661/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1661/2/testReport/ | | Max. process+thread count | 1606 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1661/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #1591: HADOOP-16629: support copyFile in s3afilesystem
steveloughran closed pull request #1591: HADOOP-16629: support copyFile in s3afilesystem URL: https://github.com/apache/hadoop/pull/1591 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16658) S3A connector does not support including the token renewer in the token identifier
[ https://issues.apache.org/jira/browse/HADOOP-16658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957984#comment-16957984 ] Hudson commented on HADOOP-16658: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17564 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17564/]) HADOOP-16658. S3A connector does not support including the token renewer (stevel: rev 1d5d7d0989e9ee2f4527dc47ba5c80e1c38f641a) * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/RoleTokenIdentifier.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/AbstractS3ATokenIdentifier.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestSessionDelegationTokens.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/SessionTokenBinding.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/FullCredentialsTokenIdentifier.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/S3ADelegationTokens.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestRoleDelegationTokens.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/TestS3ADelegationTokenSupport.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/SessionTokenIdentifier.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/AbstractDelegationTokenBinding.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/AbstractDelegationIT.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/FullCredentialsTokenBinding.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/RoleTokenBinding.java > S3A connector does not support including the token renewer in the token > identifier > -- > > Key: HADOOP-16658 > URL: https://issues.apache.org/jira/browse/HADOOP-16658 > Project: Hadoop Common > Issue Type: Bug > Components: hadoop-aws >Affects Versions: 3.3.0 >Reporter: Philip Zampino >Assignee: Philip Zampino >Priority: Major > Fix For: 3.3.0 > > > To support management of delegation token expirations by way of the Yarn > TokenRenewer facility, delegation token identifiers MUST include a valid > renewer or the associated TokenRenewer implementation will be ignored. > Currently, the renewer isn't propagated to the bindings for token creation, > which means the tokens can't ever have the renewer set on them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhjwpku opened a new pull request #1670: HDFS-14925. Rename operation should check nest snapshot
zhjwpku opened a new pull request #1670: HDFS-14925. Rename operation should check nest snapshot URL: https://github.com/apache/hadoop/pull/1670 If the src directory or any of its descendant is snapshottable and the dst directory or its any of its ancestors is snapshottable, we consider this as nested snapshot, which should be denied. Signed-off-by: Zhao Junwang ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16658) S3A connector does not support including the token renewer in the token identifier
[ https://issues.apache.org/jira/browse/HADOOP-16658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16658. - Fix Version/s: 3.3.0 Resolution: Fixed > S3A connector does not support including the token renewer in the token > identifier > -- > > Key: HADOOP-16658 > URL: https://issues.apache.org/jira/browse/HADOOP-16658 > Project: Hadoop Common > Issue Type: Bug > Components: hadoop-aws >Affects Versions: 3.3.0 >Reporter: Philip Zampino >Assignee: Philip Zampino >Priority: Major > Fix For: 3.3.0 > > > To support management of delegation token expirations by way of the Yarn > TokenRenewer facility, delegation token identifiers MUST include a valid > renewer or the associated TokenRenewer implementation will be ignored. > Currently, the renewer isn't propagated to the bindings for token creation, > which means the tokens can't ever have the renewer set on them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1664: HADOOP-16658 - S3A connector does not support including the token ren…
steveloughran commented on issue #1664: HADOOP-16658 - S3A connector does not support including the token ren… URL: https://github.com/apache/hadoop/pull/1664#issuecomment-545502822 LGTM; thanks for changes +1 and merged to trunk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #1664: HADOOP-16658 - S3A connector does not support including the token ren…
steveloughran closed pull request #1664: HADOOP-16658 - S3A connector does not support including the token ren… URL: https://github.com/apache/hadoop/pull/1664 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16658) S3A connector does not support including the token renewer in the token identifier
[ https://issues.apache.org/jira/browse/HADOOP-16658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-16658: --- Assignee: Philip Zampino > S3A connector does not support including the token renewer in the token > identifier > -- > > Key: HADOOP-16658 > URL: https://issues.apache.org/jira/browse/HADOOP-16658 > Project: Hadoop Common > Issue Type: Bug > Components: hadoop-aws >Affects Versions: 3.3.0 >Reporter: Philip Zampino >Assignee: Philip Zampino >Priority: Major > > To support management of delegation token expirations by way of the Yarn > TokenRenewer facility, delegation token identifiers MUST include a valid > renewer or the associated TokenRenewer implementation will be ignored. > Currently, the renewer isn't propagated to the bindings for token creation, > which means the tokens can't ever have the renewer set on them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16614) Missing leveldbjni package of aarch64 platform
[ https://issues.apache.org/jira/browse/HADOOP-16614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957953#comment-16957953 ] Eric Yang commented on HADOOP-16614: The patch looks good to me. I will commit if no objections. > Missing leveldbjni package of aarch64 platform > -- > > Key: HADOOP-16614 > URL: https://issues.apache.org/jira/browse/HADOOP-16614 > Project: Hadoop Common > Issue Type: Bug >Reporter: liusheng >Priority: Major > > Currently, Hadoop denpend on the *leveldbjni-all:1.8* package of > *org.fusesource.leveldbjni* group, but it cannot support ARM platform. > see: [https://search.maven.org/search?q=g:org.fusesource.leveldbjni] > Because the leveldbjni community is inactivity and the code > ([https://github.com/fusesource/leveldbjni]) didn't updated a long time.I > will build the leveldbjni package of aarch64 platform, and upload it with > other platform packages of *org.fusesource.leveldbjni* to a new > *org.openlabtesting.leveldbjni* maven repo. In hadoop code, I will add a new > profile aarch64 for for automatically select the > *org.openlabtesting.leveldbjni* artifact group and using the aarch64 package > of leveldbjni when running on ARM server, this approach has no effect on > current code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16316) S3A delegation tests fail if you set fs.s3a.secret.key
[ https://issues.apache.org/jira/browse/HADOOP-16316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16316. - Resolution: Duplicate HADOOP-16477 is the same issue; this came first but that has a PR > S3A delegation tests fail if you set fs.s3a.secret.key > -- > > Key: HADOOP-16316 > URL: https://issues.apache.org/jira/browse/HADOOP-16316 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > > The ITests for Session and Role DTs in S3A set the encryption option (to > verify its propagation). But if you have set an encryption key in the config > then test setup will fail > Fix: when you set the encryption, clear the options for fs.s3a.encryption.key > for the dest bucket -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1664: HADOOP-16658 - S3A connector does not support including the token ren…
hadoop-yetus commented on issue #1664: HADOOP-16658 - S3A connector does not support including the token ren… URL: https://github.com/apache/hadoop/pull/1664#issuecomment-545480247 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 1777 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 1119 | trunk passed | | +1 | compile | 38 | trunk passed | | +1 | checkstyle | 28 | trunk passed | | +1 | mvnsite | 40 | trunk passed | | +1 | shadedclient | 820 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 30 | trunk passed | | 0 | spotbugs | 60 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 59 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 33 | the patch passed | | +1 | compile | 29 | the patch passed | | +1 | javac | 29 | the patch passed | | -0 | checkstyle | 20 | hadoop-tools/hadoop-aws: The patch generated 3 new + 9 unchanged - 0 fixed = 12 total (was 9) | | +1 | mvnsite | 33 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 792 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 26 | the patch passed | | +1 | findbugs | 70 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 86 | hadoop-aws in the patch passed. | | +1 | asflicense | 33 | The patch does not generate ASF License warnings. | | | | 5119 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1664/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1664 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux f23e524c59ac 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / a901405 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1664/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1664/2/testReport/ | | Max. process+thread count | 472 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1664/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on issue #1661: HADOOP-16484. S3A to warn or fail if S3Guard is disabled
bgaborg commented on issue #1661: HADOOP-16484. S3A to warn or fail if S3Guard is disabled URL: https://github.com/apache/hadoop/pull/1661#issuecomment-545472813 The last commit contains fixes based on the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] pzampino commented on issue #1664: HADOOP-16658 - S3A connector does not support including the token ren…
pzampino commented on issue #1664: HADOOP-16658 - S3A connector does not support including the token ren… URL: https://github.com/apache/hadoop/pull/1664#issuecomment-545440963 I've made the suggested changes, and re-ran the tests (which passed again). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16656) Document FairCallQueue configs in core-default.xml
[
https://issues.apache.org/jira/browse/HADOOP-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957846#comment-16957846
]
Dinesh Chitlangia commented on HADOOP-16656:
[~smeng] Thank you for checking. Your suggestion sounds fair to me. Let us just
remind users that they typically want to set {{[port_number]}} to NN RPC port
rather than the other two (NN service RPC port, or lifeline port).
> Document FairCallQueue configs in core-default.xml
> --
>
> Key: HADOOP-16656
> URL: https://issues.apache.org/jira/browse/HADOOP-16656
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16656.001.patch
>
>
> So far those callqueue / scheduler / faircallqueue -related configurations
> are only documented in FairCallQueue.md in 3.3.0:
> https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-project-dist/hadoop-common/FairCallQueue.html#Full_List_of_Configurations
> (Thanks Akira for uploading this.)
> Goal: Document those configs in core-default.xml as well to make it easier
> for users(admins) to find and use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16657) Move remaining log4j APIs over to slf4j in hadoop-common.
[ https://issues.apache.org/jira/browse/HADOOP-16657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957803#comment-16957803 ] Minni Mittal edited comment on HADOOP-16657 at 10/23/19 11:57 AM: -- Following files in hadoop-common project are using log4j's API and IOUtils.cleanuo() instead of slf4j and IOUtils.CleanUpwithLogger() respectively: 1. NflyFSystem.java 2. TestBloomMapFile.java 3. TestMapFile.java 4. Server.java 5. DikChecker.java was (Author: minni31): Following files in hadoop-common project are using log4j's logger and IOUtils.cleanuo() instead of slf4j and IOUtils.CleanUpwithLogger() respectively: 1. NflyFSystem.java 2. TestBloomMapFile.java 3. TestMapFile.java 4. Server.java 5. DikChecker.java > Move remaining log4j APIs over to slf4j in hadoop-common. > - > > Key: HADOOP-16657 > URL: https://issues.apache.org/jira/browse/HADOOP-16657 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Minni Mittal >Priority: Major > Attachments: HADOOP-16657.001.patch > > > There are some remaining classes where log4j's APIs are still being used. > Created this Jira to move them to log4j2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16657) Move remaining log4j APIs over to slf4j in hadoop-common.
[ https://issues.apache.org/jira/browse/HADOOP-16657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957803#comment-16957803 ] Minni Mittal edited comment on HADOOP-16657 at 10/23/19 11:56 AM: -- Following files in hadoop-common project are using log4j's logger and IOUtils.cleanuo() instead of slf4j and IOUtils.CleanUpwithLogger() respectively: 1. NflyFSystem.java 2. TestBloomMapFile.java 3. TestMapFile.java 4. Server.java 5. DikChecker.java was (Author: minni31): Following files in hadoop-common project are modified for slf4j support: 1. NflyFSystem.java 2. TestBloomMapFile.java 3. TestMapFile.java 4. Server.java 5. DikChecker.java > Move remaining log4j APIs over to slf4j in hadoop-common. > - > > Key: HADOOP-16657 > URL: https://issues.apache.org/jira/browse/HADOOP-16657 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Minni Mittal >Priority: Major > Attachments: HADOOP-16657.001.patch > > > There are some remaining classes where log4j's APIs are still being used. > Created this Jira to move them to log4j2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16657) Move remaining log4j APIs over to slf4j in hadoop-common.
[ https://issues.apache.org/jira/browse/HADOOP-16657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957803#comment-16957803 ] Minni Mittal commented on HADOOP-16657: --- Following files in hadoop-common project are modified for slf4j support: 1. NflyFSystem.java 2. TestBloomMapFile.java 3. TestMapFile.java 4. Server.java 5. DikChecker.java > Move remaining log4j APIs over to slf4j in hadoop-common. > - > > Key: HADOOP-16657 > URL: https://issues.apache.org/jira/browse/HADOOP-16657 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Minni Mittal >Priority: Major > Attachments: HADOOP-16657.001.patch > > > There are some remaining classes where log4j's APIs are still being used. > Created this Jira to move them to log4j2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16657) Move remaining log4j APIs over to slf4j in hadoop-common.
[ https://issues.apache.org/jira/browse/HADOOP-16657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Minni Mittal updated HADOOP-16657: -- Attachment: HADOOP-16657.001.patch > Move remaining log4j APIs over to slf4j in hadoop-common. > - > > Key: HADOOP-16657 > URL: https://issues.apache.org/jira/browse/HADOOP-16657 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Minni Mittal >Priority: Major > Attachments: HADOOP-16657.001.patch > > > There are some remaining classes where log4j's APIs are still being used. > Created this Jira to move them to log4j2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1667: HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuff…
hadoop-yetus commented on issue #1667: HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuff… URL: https://github.com/apache/hadoop/pull/1667#issuecomment-545340879 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 35 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 14 | Maven dependency ordering for branch | | +1 | mvninstall | 1091 | trunk passed | | +1 | compile | 213 | trunk passed | | +1 | checkstyle | 56 | trunk passed | | +1 | mvnsite | 127 | trunk passed | | +1 | shadedclient | 934 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 117 | trunk passed | | 0 | spotbugs | 168 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 296 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 15 | Maven dependency ordering for patch | | +1 | mvninstall | 110 | the patch passed | | +1 | compile | 194 | the patch passed | | +1 | javac | 194 | the patch passed | | -0 | checkstyle | 50 | hadoop-hdfs-project: The patch generated 13 new + 1 unchanged - 0 fixed = 14 total (was 1) | | +1 | mvnsite | 114 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 784 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 113 | the patch passed | | +1 | findbugs | 312 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 128 | hadoop-hdfs-client in the patch passed. | | -1 | unit | 5199 | hadoop-hdfs in the patch failed. | | +1 | asflicense | 42 | The patch does not generate ASF License warnings. | | | | 9984 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController | | | hadoop.hdfs.TestMultipleNNPortQOP | | | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1667/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1667 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 5aa49b74bf0b 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / a901405 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1667/3/artifact/out/diff-checkstyle-hadoop-hdfs-project.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1667/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1667/3/testReport/ | | Max. process+thread count | 3985 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1667/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16656) Document FairCallQueue configs in core-default.xml
[
https://issues.apache.org/jira/browse/HADOOP-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957628#comment-16957628
]
Siyao Meng commented on HADOOP-16656:
-
[~dineshchitlangia] Thanks for the comment.
Yes, in the most typical usage of FairCallQueue, {{[port_number]}} should be
set to the NN RPC port (in {{dfs.namenode.rpc-address}}). But I don't really
see anything preventing users from setting {{[port_number]}} to NN service RPC
port (in dfs.namenode.servicerpc-address) or lifeline port (in
{{dfs.namenode.lifeline.rpc-address}}).
So I tested (trunk) locally with 1 NN and 1 DN,
(1) {{[port_number]}} can be set to NN RPC port (in
{{dfs.namenode.rpc-address}}) *without* setting a NN service RPC address/port
({{dfs.namenode.servicerpc-address}});
(2) {{[port_number]}} can be set to NN service RPC port (in
{{dfs.namenode.servicerpc-address}});
(3) {{[port_number]}} can be set to NN lifeline RPC port (in
{{dfs.namenode.lifeline.rpc-address}});
In all 3 cases above, I can start NN and DN, and successfully run a shell
command "{{hdfs dfs -ls /}}" with the NN.
One explanation is that, even if {{ipc.[port_number].callqueue.impl}} is *not*
set, it defaults to {{LinkedBlockingQueue}}, as shown in the code
[here|https://github.com/apache/hadoop/blob/1654497f98fb7f2de8214d2fbad305b7a2854816/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java#L691].
So in theory, {{[port_number]}} could be set to any RPC port opened on NN.
I haven't dug into FairCallQueue implementation in its entirety - there might
be some other exceptions I didn't realize.
Instead, we could remind users that they typically want to set
{{[port_number]}} to NN RPC port rather than the other two (NN service RPC
port, or lifeline port)?
> Document FairCallQueue configs in core-default.xml
> --
>
> Key: HADOOP-16656
> URL: https://issues.apache.org/jira/browse/HADOOP-16656
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16656.001.patch
>
>
> So far those callqueue / scheduler / faircallqueue -related configurations
> are only documented in FairCallQueue.md in 3.3.0:
> https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-project-dist/hadoop-common/FairCallQueue.html#Full_List_of_Configurations
> (Thanks Akira for uploading this.)
> Goal: Document those configs in core-default.xml as well to make it easier
> for users(admins) to find and use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
