[GitHub] [hadoop] hadoop-yetus commented on pull request #4258: YARN-11125. Backport YARN-6483 to branch-2.10
hadoop-yetus commented on PR #4258:
URL: https://github.com/apache/hadoop/pull/4258#issuecomment-1115739795
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 10 new or modified test files. |
_ branch-2.10 Compile Tests _ |
| +0 :ok: | mvndep | 4m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 14m 16s | | branch-2.10 passed |
| +1 :green_heart: | compile | 7m 28s | | branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | compile | 6m 37s | | branch-2.10 passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | checkstyle | 1m 37s | | branch-2.10 passed |
| +1 :green_heart: | mvnsite | 6m 12s | | branch-2.10 passed |
| +1 :green_heart: | javadoc | 5m 49s | | branch-2.10 passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 5m 9s | | branch-2.10 passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | spotbugs | 9m 26s | | branch-2.10 passed |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 10s | | the patch passed |
| +1 :green_heart: | compile | 6m 42s | | the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | cc | 6m 42s | | the patch passed |
| -1 :x: | javac | 6m 42s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkAzulSystems,Inc.-1.7.0_262-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4258/1/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkAzulSystems,Inc.-1.7.0_262-b10.txt)
| hadoop-yarn-project_hadoop-yarn-jdkAzulSystems,Inc.-1.7.0_262-b10 with JDK
Azul Systems, Inc.-1.7.0_262-b10 generated 4 new + 141 unchanged - 0 fixed =
145 total (was 141) |
| +1 :green_heart: | compile | 6m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | cc | 6m 32s | | the patch passed |
| -1 :x: | javac | 6m 32s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4258/1/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07.txt)
|
hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 generated 4 new
+ 130 unchanged - 0 fixed = 134 total (was 130) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 23s | |
hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 487 unchanged - 19
fixed = 487 total (was 506) |
| +1 :green_heart: | mvnsite | 5m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 5s | | the patch passed with JDK
Azul Systems, Inc.-1.7.0_262-b10 |
| +1 :green_heart: | javadoc | 4m 38s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07 |
| +1 :green_heart: | spotbugs | 9m 23s | | the patch passed |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 11s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 3m 49s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 56s | | hadoop-yarn-server-common in
the patch passed. |
| -1 :x: | unit | 68m 57s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4258/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 24m 9s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 13s | | The patch does not
generate ASF License warnings. |
| | | 226m 6s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4252: HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits
hadoop-yetus commented on PR #4252:
URL: https://github.com/apache/hadoop/pull/4252#issuecomment-1115722396
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 6s | | trunk passed |
| +1 :green_heart: | compile | 6m 26s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 6m 2s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 26s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 53s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 19s | | the patch passed |
| +1 :green_heart: | compile | 6m 2s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 6m 2s | | the patch passed |
| +1 :green_heart: | javac | 6m 2s | | the patch passed |
| +1 :green_heart: | compile | 5m 48s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 5m 48s | | the patch passed |
| +1 :green_heart: | javac | 5m 48s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 17s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4252/7/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 3 new + 209 unchanged - 0 fixed =
212 total (was 209) |
| +1 :green_heart: | mvnsite | 2m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 47s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 17s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 37s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 248m 29s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4252/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not
generate ASF License warnings. |
| | | 396m 18s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.cli.TestHDFSCLI |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4252/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4252 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit buflint
bufcompat codespell javadoc mvninstall shadedclient spotbugs checkstyle |
| uname | Linux 129385a8da2c 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 85673d1c65ea185d277f030b4c4e2c55ffe3cf42 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
[GitHub] [hadoop] iwasakims commented on a diff in pull request #4247: MAPREDUCE-7369. Fixed MapReduce tasks timing out when spends more time on MultipleOutputs#close
iwasakims commented on code in PR #4247: URL: https://github.com/apache/hadoop/pull/4247#discussion_r863342147 ## hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml: ## @@ -286,6 +286,13 @@ + + mapreduce.task.enable.ping-for-liveliness-check + true Review Comment: The value must be same with the DEFAULT_MR_TASK_ENABLE_PING_FOR_LIVELINESS_CHECK. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #4209: HDFS-16550. [SBN read] Improper cache-size for journal node may cause cluster crash
tomscut commented on PR #4209: URL: https://github.com/apache/hadoop/pull/4209#issuecomment-1115568961 > I think this change is a bit too restrictive. There may well be valid use cases for setting it above the 90% threshold. For example if you configured a 100GB heap, you really don't need 10GB of non-cache overhead, so you could safely allocate 95GB for the cache. > > If we want to add fail-fast behavior, I would say it should only apply when `cache size >= heap size`. This is clearly invalid -- you need at least _some_ overhead heap memory. > > Alternatively, you could make the 90% threshold configurable, and point users to a config they can adjust if they really want to exceed it. But I think this may be overkill. Thanks @xkrogen for review and comments. Maybe we can do this: Do not set the `cache size` to a fixed value, but to the ratio of maximum memory, which is 0.2 by default. This avoids the problem of too large cache size. In addition, users can actively adjust the heap size when they need to increase the cache size. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
virajjasani commented on PR #4107: URL: https://github.com/apache/hadoop/pull/4107#issuecomment-1115523698 Thanks everyone for the reviews, here is the branch-3.3 backport PR #4259 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani opened a new pull request, #4259: HDFS-16521. DFS API to retrieve slow datanodes (#4107)
virajjasani opened a new pull request, #4259: URL: https://github.com/apache/hadoop/pull/4259 branch-3.3 backport PR of #4107 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ashutoshcipher opened a new pull request, #4258: YARN-11125. Backport YARN-6483 to branch-2.10
ashutoshcipher opened a new pull request, #4258: URL: https://github.com/apache/hadoop/pull/4258 ### Description of PR Backport YARN-6483 to branch-2.10 * JIRA: YARN-11125 - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ashutoshcipher commented on a diff in pull request #4244: YARN-11119. Backport YARN-10538 to branch-2.10
ashutoshcipher commented on code in PR #4244:
URL: https://github.com/apache/hadoop/pull/4244#discussion_r863294688
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCNodeUpdates.java:
##
@@ -98,6 +111,47 @@ public AllocateResponse run() throws Exception {
});
}
+ @Test
+ public void testAMRMRecommissioningNodes() throws Exception {
+MockNM nm1 = rm.registerNode("127.0.0.1:1234", 1);
+MockNM nm2 = rm.registerNode("127.0.0.2:1234", 1);
+rm.drainEvents();
+
+RMApp app1 = rm.submitApp(2000);
+
+// Trigger the scheduling so the AM gets 'launched' on nm1
+nm1.nodeHeartbeat(true);
+
+RMAppAttempt attempt1 = app1.getCurrentAppAttempt();
+MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId());
+
+// register AM returns no unusable node
+am1.registerAppAttempt();
+
+// DECOMMISSION nm2
+syncNodeGracefulDecommission(nm2);
+
+AllocateRequest allocateRequest1 =
+AllocateRequest.newInstance(0, 0F, null, null, null);
+AllocateResponse response1 =
+allocate(attempt1.getAppAttemptId(), allocateRequest1);
+List updatedNodes = response1.getUpdatedNodes();
+Assert.assertEquals(0, updatedNodes.size());
Review Comment:
Yeah @aajisaka - this makes sense. Let's try back porting
[YARN-6483](https://issues.apache.org/jira/browse/YARN-6483) and see if there
aren't much dependencies to be backported along with it and we can successfully
do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4257: MAPREDUCE-7376. AggregateWordCount fetches wrong results.
hadoop-yetus commented on PR #4257:
URL: https://github.com/apache/hadoop/pull/4257#issuecomment-1115499005
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 18s | | trunk passed |
| +1 :green_heart: | compile | 3m 1s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 2m 32s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 21s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 4s | | the patch passed |
| +1 :green_heart: | compile | 2m 48s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 2m 48s | | the patch passed |
| +1 :green_heart: | compile | 2m 20s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 2m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 2s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 7m 12s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 1m 2s | | hadoop-mapreduce-examples in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 130m 39s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4257/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4257 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 6d5c1dac71e1 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 79e21fdae925377c020fc5afcb8079fdb08d94bc |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4257/3/testReport/ |
| Max. process+thread count | 1074 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-examples U: hadoop-mapreduce-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4257/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was
[GitHub] [hadoop] tomscut commented on a diff in pull request #4201: HDFS-16547. [SBN read] Namenode in safe mode should not be transfered to observer state
tomscut commented on code in PR #4201:
URL: https://github.com/apache/hadoop/pull/4201#discussion_r863251873
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -1899,6 +1899,10 @@ synchronized void transitionToStandby() throws
IOException {
synchronized void transitionToObserver() throws IOException {
String operationName = "transitionToObserver";
namesystem.checkSuperuserPrivilege(operationName);
+if (namesystem.isInSafeMode()) {
Review Comment:
Thank you @xkrogen very much for the review and careful suggestions.
The intent here is really the same as
`dfs.ha.nn.not-become-active-in-safemode`. It's just that the configuration
name looks a bit conflicting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a diff in pull request #4201: HDFS-16547. [SBN read] Namenode in safe mode should not be transfered to observer state
tomscut commented on code in PR #4201:
URL: https://github.com/apache/hadoop/pull/4201#discussion_r863251873
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -1899,6 +1899,10 @@ synchronized void transitionToStandby() throws
IOException {
synchronized void transitionToObserver() throws IOException {
String operationName = "transitionToObserver";
namesystem.checkSuperuserPrivilege(operationName);
+if (namesystem.isInSafeMode()) {
Review Comment:
Thank you @xkrogen very much for the review and careful suggestions.
The intent here is really the same as
`dfs.ha.nn.not-become-active-in-safemode`. It's just that the configuration
name looks a bit conflicting.
A more appropriate name would be
dfs.ha.nn.do.not-accept-requests-in-safemode. Maybe we can change it later.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -1899,6 +1899,10 @@ synchronized void transitionToStandby() throws
IOException {
synchronized void transitionToObserver() throws IOException {
String operationName = "transitionToObserver";
namesystem.checkSuperuserPrivilege(operationName);
+if (namesystem.isInSafeMode()) {
Review Comment:
Thank you @xkrogen very much for the review and careful suggestions.
The intent here is really the same as
`dfs.ha.nn.not-become-active-in-safemode`. It's just that the configuration
name looks a bit conflicting.
A more appropriate name would be
`dfs.ha.nn.do.not-accept-requests-in-safemode`. Maybe we can change it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17890) ABFS: Refactor HTTP request handling code
[
https://issues.apache.org/jira/browse/HADOOP-17890?focusedWorklogId=765218=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-765218
]
ASF GitHub Bot logged work on HADOOP-17890:
---
Author: ASF GitHub Bot
Created on: 02/May/22 23:49
Start Date: 02/May/22 23:49
Worklog Time Spent: 10m
Work Description: raymondlam12 commented on code in PR #3381:
URL: https://github.com/apache/hadoop/pull/3381#discussion_r863255244
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java:
##
@@ -336,95 +347,141 @@ public void sendRequest(byte[] buffer, int offset, int
length) throws IOExceptio
*
* @throws IOException if an error occurs.
*/
- public void processResponse(final byte[] buffer, final int offset, final int
length) throws IOException {
+ public void processResponse(final byte[] buffer,
+ final int offset,
+ final int length) throws IOException {
// get the response
long startTime = 0;
-if (this.isTraceEnabled) {
+if (isTraceEnabled) {
startTime = System.nanoTime();
}
-this.statusCode = this.connection.getResponseCode();
+statusCode = connection.getResponseCode();
-if (this.isTraceEnabled) {
- this.recvResponseTimeMs = elapsedTimeMs(startTime);
+if (isTraceEnabled) {
+ recvResponseTimeMs = elapsedTimeMs(startTime);
}
-this.statusDescription = this.connection.getResponseMessage();
+statusDescription = connection.getResponseMessage();
-this.requestId =
this.connection.getHeaderField(HttpHeaderConfigurations.X_MS_REQUEST_ID);
-if (this.requestId == null) {
- this.requestId = AbfsHttpConstants.EMPTY_STRING;
+requestId =
connection.getHeaderField(HttpHeaderConfigurations.X_MS_REQUEST_ID);
+if (requestId == null) {
+ requestId = AbfsHttpConstants.EMPTY_STRING;
}
// dump the headers
AbfsIoUtils.dumpHeadersToDebugLog("Response Headers",
connection.getHeaderFields());
-if (AbfsHttpConstants.HTTP_METHOD_HEAD.equals(this.method)) {
+if (AbfsHttpConstants.HTTP_METHOD_HEAD.equals(method)) {
// If it is HEAD, and it is ERROR
return;
}
-if (this.isTraceEnabled) {
+if (isTraceEnabled) {
startTime = System.nanoTime();
}
+long totalBytesRead = 0;
+
+try {
+ totalBytesRead = parseResponse(buffer, offset, length);
+} finally {
+ if (isTraceEnabled) {
+recvResponseTimeMs += elapsedTimeMs(startTime);
+ }
+ bytesReceived = totalBytesRead;
+}
+ }
+
+ /**
+ * Detects if the Http response indicates an error or success response.
+ * Parses the response and returns the number of bytes read from the
+ * response.
+ *
+ * @param buffer a buffer to hold the response entity body.
+ * @param offset an offset in the buffer where the data will being.
+ * @param length the number of bytes to be written to the buffer.
+ * @return number of bytes read from response InputStream.
+ * @throws IOException if an error occurs.
+ */
+ public long parseResponse(final byte[] buffer,
+ final int offset,
+ final int length) throws IOException {
if (statusCode >= HttpURLConnection.HTTP_BAD_REQUEST) {
processStorageErrorResponse();
- if (this.isTraceEnabled) {
-this.recvResponseTimeMs += elapsedTimeMs(startTime);
- }
- this.bytesReceived =
this.connection.getHeaderFieldLong(HttpHeaderConfigurations.CONTENT_LENGTH, 0);
+ return connection.getHeaderFieldLong(
+ HttpHeaderConfigurations.CONTENT_LENGTH, 0);
} else {
- // consume the input stream to release resources
- int totalBytesRead = 0;
-
- try (InputStream stream = this.connection.getInputStream()) {
+ try (InputStream stream = connection.getInputStream()) {
if (isNullInputStream(stream)) {
- return;
+ return 0;
}
-boolean endOfStream = false;
-// this is a list operation and need to retrieve the data
-// need a better solution
-if (AbfsHttpConstants.HTTP_METHOD_GET.equals(this.method) && buffer ==
null) {
+// Incase of ListStatus call, request is of GET Method and the
+// caller doesnt provide buffer because the length can not be
+// pre-determined
+if (AbfsHttpConstants.HTTP_METHOD_GET.equals(method)
+&& buffer == null) {
parseListFilesResponse(stream);
} else {
- if (buffer != null) {
-while (totalBytesRead < length) {
- int bytesRead = stream.read(buffer, offset + totalBytesRead,
length - totalBytesRead);
- if (bytesRead == -1) {
-endOfStream = true;
-break;
- }
- totalBytesRead += bytesRead;
-}
- }
-
[GitHub] [hadoop] raymondlam12 commented on a diff in pull request #3381: HADOOP-17890. ABFS: Http request handling code refactoring
raymondlam12 commented on code in PR #3381:
URL: https://github.com/apache/hadoop/pull/3381#discussion_r863255244
##
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java:
##
@@ -336,95 +347,141 @@ public void sendRequest(byte[] buffer, int offset, int
length) throws IOExceptio
*
* @throws IOException if an error occurs.
*/
- public void processResponse(final byte[] buffer, final int offset, final int
length) throws IOException {
+ public void processResponse(final byte[] buffer,
+ final int offset,
+ final int length) throws IOException {
// get the response
long startTime = 0;
-if (this.isTraceEnabled) {
+if (isTraceEnabled) {
startTime = System.nanoTime();
}
-this.statusCode = this.connection.getResponseCode();
+statusCode = connection.getResponseCode();
-if (this.isTraceEnabled) {
- this.recvResponseTimeMs = elapsedTimeMs(startTime);
+if (isTraceEnabled) {
+ recvResponseTimeMs = elapsedTimeMs(startTime);
}
-this.statusDescription = this.connection.getResponseMessage();
+statusDescription = connection.getResponseMessage();
-this.requestId =
this.connection.getHeaderField(HttpHeaderConfigurations.X_MS_REQUEST_ID);
-if (this.requestId == null) {
- this.requestId = AbfsHttpConstants.EMPTY_STRING;
+requestId =
connection.getHeaderField(HttpHeaderConfigurations.X_MS_REQUEST_ID);
+if (requestId == null) {
+ requestId = AbfsHttpConstants.EMPTY_STRING;
}
// dump the headers
AbfsIoUtils.dumpHeadersToDebugLog("Response Headers",
connection.getHeaderFields());
-if (AbfsHttpConstants.HTTP_METHOD_HEAD.equals(this.method)) {
+if (AbfsHttpConstants.HTTP_METHOD_HEAD.equals(method)) {
// If it is HEAD, and it is ERROR
return;
}
-if (this.isTraceEnabled) {
+if (isTraceEnabled) {
startTime = System.nanoTime();
}
+long totalBytesRead = 0;
+
+try {
+ totalBytesRead = parseResponse(buffer, offset, length);
+} finally {
+ if (isTraceEnabled) {
+recvResponseTimeMs += elapsedTimeMs(startTime);
+ }
+ bytesReceived = totalBytesRead;
+}
+ }
+
+ /**
+ * Detects if the Http response indicates an error or success response.
+ * Parses the response and returns the number of bytes read from the
+ * response.
+ *
+ * @param buffer a buffer to hold the response entity body.
+ * @param offset an offset in the buffer where the data will being.
+ * @param length the number of bytes to be written to the buffer.
+ * @return number of bytes read from response InputStream.
+ * @throws IOException if an error occurs.
+ */
+ public long parseResponse(final byte[] buffer,
+ final int offset,
+ final int length) throws IOException {
if (statusCode >= HttpURLConnection.HTTP_BAD_REQUEST) {
processStorageErrorResponse();
- if (this.isTraceEnabled) {
-this.recvResponseTimeMs += elapsedTimeMs(startTime);
- }
- this.bytesReceived =
this.connection.getHeaderFieldLong(HttpHeaderConfigurations.CONTENT_LENGTH, 0);
+ return connection.getHeaderFieldLong(
+ HttpHeaderConfigurations.CONTENT_LENGTH, 0);
} else {
- // consume the input stream to release resources
- int totalBytesRead = 0;
-
- try (InputStream stream = this.connection.getInputStream()) {
+ try (InputStream stream = connection.getInputStream()) {
if (isNullInputStream(stream)) {
- return;
+ return 0;
}
-boolean endOfStream = false;
-// this is a list operation and need to retrieve the data
-// need a better solution
-if (AbfsHttpConstants.HTTP_METHOD_GET.equals(this.method) && buffer ==
null) {
+// Incase of ListStatus call, request is of GET Method and the
+// caller doesnt provide buffer because the length can not be
+// pre-determined
+if (AbfsHttpConstants.HTTP_METHOD_GET.equals(method)
+&& buffer == null) {
parseListFilesResponse(stream);
} else {
- if (buffer != null) {
-while (totalBytesRead < length) {
- int bytesRead = stream.read(buffer, offset + totalBytesRead,
length - totalBytesRead);
- if (bytesRead == -1) {
-endOfStream = true;
-break;
- }
- totalBytesRead += bytesRead;
-}
- }
- if (!endOfStream && stream.read() != -1) {
-// read and discard
-int bytesRead = 0;
-byte[] b = new byte[CLEAN_UP_BUFFER_SIZE];
-while ((bytesRead = stream.read(b)) >= 0) {
- totalBytesRead += bytesRead;
-}
- }
+ return readDataFromStream(stream, buffer, offset, length);
}
- } catch (IOException ex) {
-
[GitHub] [hadoop] tomscut commented on a diff in pull request #4201: HDFS-16547. [SBN read] Namenode in safe mode should not be transfered to observer state
tomscut commented on code in PR #4201:
URL: https://github.com/apache/hadoop/pull/4201#discussion_r863251873
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -1899,6 +1899,10 @@ synchronized void transitionToStandby() throws
IOException {
synchronized void transitionToObserver() throws IOException {
String operationName = "transitionToObserver";
namesystem.checkSuperuserPrivilege(operationName);
+if (namesystem.isInSafeMode()) {
Review Comment:
Thank you @xkrogen very much for the review and careful suggestions.
The intent here is really the same as
`dfs.ha.nn.not-become-active-in-safemode`. It's just that the configuration
name looks a bit conflicting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4257: MAPREDUCE-7376. AggregateWordCount fetches wrong results.
hadoop-yetus commented on PR #4257:
URL: https://github.com/apache/hadoop/pull/4257#issuecomment-1115416677
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 41s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 28m 52s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4257/2/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 3m 16s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 2m 39s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 21s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 11s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 4s | | the patch passed |
| +1 :green_heart: | compile | 2m 48s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 2m 48s | | the patch passed |
| +1 :green_heart: | compile | 2m 21s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 2m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 2s |
[/results-checkstyle-hadoop-mapreduce-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4257/2/artifact/out/results-checkstyle-hadoop-mapreduce-project.txt)
| hadoop-mapreduce-project: The patch generated 1 new + 9 unchanged - 0 fixed
= 10 total (was 9) |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 11s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 6m 22s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 0m 52s | | hadoop-mapreduce-examples in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 129m 2s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4257/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4257 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux ab49bff137fa 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8707ebc259abf74e5928f847051b966562cafc33 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4257/2/testReport/ |
| Max. process+thread count | 1102 (vs. ulimit of 5500) |
| modules | C:
[GitHub] [hadoop] RuinanGu commented on a diff in pull request #4252: HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits
RuinanGu commented on code in PR #4252:
URL: https://github.com/apache/hadoop/pull/4252#discussion_r863162303
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructionInfo.java:
##
@@ -41,26 +41,28 @@ public class StripedReconstructionInfo {
private final DatanodeInfo[] targets;
private final StorageType[] targetStorageTypes;
private final String[] targetStorageIds;
+ private final byte[] excludeReconstructedIndices;
public StripedReconstructionInfo(ExtendedBlock blockGroup,
ErasureCodingPolicy ecPolicy, byte[] liveIndices, DatanodeInfo[] sources,
byte[] targetIndices) {
this(blockGroup, ecPolicy, liveIndices, sources, targetIndices, null,
-null, null);
+null, null, new byte[0]);
}
StripedReconstructionInfo(ExtendedBlock blockGroup,
ErasureCodingPolicy ecPolicy, byte[] liveIndices, DatanodeInfo[] sources,
DatanodeInfo[] targets, StorageType[] targetStorageTypes,
- String[] targetStorageIds) {
+ String[] targetStorageIds, byte[] excludeReconstructedIndices) {
Review Comment:
When the DN receive the command of EC reconstruction, we can get
BlockECReconstructionCommand->
BlockECReconstructionInfo->StripedReconstructionInfo->StripedBlockReconstructor->StripedWriter,
ErasureCodingWorker.processErasureCodingTasks() calls this constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] RuinanGu commented on a diff in pull request #4252: HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits
RuinanGu commented on code in PR #4252:
URL: https://github.com/apache/hadoop/pull/4252#discussion_r863162303
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructionInfo.java:
##
@@ -41,26 +41,28 @@ public class StripedReconstructionInfo {
private final DatanodeInfo[] targets;
private final StorageType[] targetStorageTypes;
private final String[] targetStorageIds;
+ private final byte[] excludeReconstructedIndices;
public StripedReconstructionInfo(ExtendedBlock blockGroup,
ErasureCodingPolicy ecPolicy, byte[] liveIndices, DatanodeInfo[] sources,
byte[] targetIndices) {
this(blockGroup, ecPolicy, liveIndices, sources, targetIndices, null,
-null, null);
+null, null, new byte[0]);
}
StripedReconstructionInfo(ExtendedBlock blockGroup,
ErasureCodingPolicy ecPolicy, byte[] liveIndices, DatanodeInfo[] sources,
DatanodeInfo[] targets, StorageType[] targetStorageTypes,
- String[] targetStorageIds) {
+ String[] targetStorageIds, byte[] excludeReconstructedIndices) {
Review Comment:
When the DN receive the command of EC reconstruction, we can get
BlockECReconstructionCommand->
BlockECReconstructionInfo->StripedReconstructionInfo->StripedBlockReconstructor->StripedWriter,
ErasureCodingWorker.processErasureCodingTasks() calls this constructor
function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18193) Support nested mount points in INodeTree
[ https://issues.apache.org/jira/browse/HADOOP-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei Yang updated HADOOP-18193: -- Attachment: Nested Mount Point in ViewFs.pdf > Support nested mount points in INodeTree > > > Key: HADOOP-18193 > URL: https://issues.apache.org/jira/browse/HADOOP-18193 > Project: Hadoop Common > Issue Type: Improvement > Components: viewfs >Affects Versions: 2.10.0 >Reporter: Lei Yang >Assignee: Lei Yang >Priority: Major > Labels: pull-request-available > Attachments: Nested Mount Point in ViewFs.pdf > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Defining following client mount table config is not supported in INodeTree > and will throw FileAlreadyExistsException > fs.viewfs.mounttable.link./foo/bar=hdfs://nn1/foo/bar > fs.viewfs.mounttable.link./foo=hdfs://nn02/foo > > INodeTree has 2 methods that need change to support nested mount points. > createLink(..): build INodeTree during fs init. > resolve(..): resolve path in INodeTree with viewfs apis. > > ViewFileSystem and ViewFs referes INodeTree.resolve(..) to resolve path to > specific mount point. No changes are expected in both classes. However, we > need to support existing use cases and make sure no regression are caused. > > AC: > # INodeTree.createlink should support creating nested mount > points.(INodeTree is constructed during fs init) > # INodeTree.resolve should support resolve path based on nested mount > points. (INodeTree.resolve is used in viewfs apis) > # No regression in existing ViewFileSystem and ViewFs apis. > # Ensure some important apis are not broken with nested mount points. > (Rename, getContentSummary, listStatus...) -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang merged pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
jojochuang merged PR #4107: URL: https://github.com/apache/hadoop/pull/4107 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4257: MAPREDUCE-7376. AggregateWordCount fetches wrong results.
hadoop-yetus commented on PR #4257:
URL: https://github.com/apache/hadoop/pull/4257#issuecomment-1115360668
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 32s | | trunk passed |
| +1 :green_heart: | compile | 0m 54s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 51s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 33s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 47s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 6m 46s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 113m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4257/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4257 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux b591acbe67cf 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4a3ec5965d23831c5539a0bfda1753607846457b |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4257/1/testReport/ |
| Max. process+thread count | 894 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4257/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on
[GitHub] [hadoop] jojochuang commented on a diff in pull request #4252: HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits
jojochuang commented on code in PR #4252:
URL: https://github.com/apache/hadoop/pull/4252#discussion_r863164865
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/erasurecoding.proto:
##
@@ -107,7 +107,8 @@ message BlockECReconstructionInfoProto {
required StorageUuidsProto targetStorageUuids = 4;
required StorageTypesProto targetStorageTypes = 5;
required bytes liveBlockIndices = 6;
- required ErasureCodingPolicyProto ecPolicy = 7;
+ required bytes excludeReconstructedIndices = 7;
Review Comment:
If ecPolicy was 7, you can't assign 7 to excludeReconstructedIndices. Please
assign it as 8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] RuinanGu commented on a diff in pull request #4252: HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits
RuinanGu commented on code in PR #4252:
URL: https://github.com/apache/hadoop/pull/4252#discussion_r863162214
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/erasurecoding.proto:
##
@@ -107,7 +107,8 @@ message BlockECReconstructionInfoProto {
required StorageUuidsProto targetStorageUuids = 4;
required StorageTypesProto targetStorageTypes = 5;
required bytes liveBlockIndices = 6;
- required ErasureCodingPolicyProto ecPolicy = 7;
+ required bytes excludeReconstructedIndices = 7;
Review Comment:
Thanks for review! Should I respect the order of the old parameters and add
the new ones to the last to make it compatible?
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructionInfo.java:
##
@@ -41,26 +41,28 @@ public class StripedReconstructionInfo {
private final DatanodeInfo[] targets;
private final StorageType[] targetStorageTypes;
private final String[] targetStorageIds;
+ private final byte[] excludeReconstructedIndices;
public StripedReconstructionInfo(ExtendedBlock blockGroup,
ErasureCodingPolicy ecPolicy, byte[] liveIndices, DatanodeInfo[] sources,
byte[] targetIndices) {
this(blockGroup, ecPolicy, liveIndices, sources, targetIndices, null,
-null, null);
+null, null, new byte[0]);
}
StripedReconstructionInfo(ExtendedBlock blockGroup,
ErasureCodingPolicy ecPolicy, byte[] liveIndices, DatanodeInfo[] sources,
DatanodeInfo[] targets, StorageType[] targetStorageTypes,
- String[] targetStorageIds) {
+ String[] targetStorageIds, byte[] excludeReconstructedIndices) {
Review Comment:
When the DN receive the command of EC reconstruction, we can get
BlockECReconstructionCommand->
BlockECReconstructionInfo->StripedReconstructionInfo->StripedBlockReconstructor->StripedWriter,
and StripedWriter.initTargetIndices() need to be modified to exclude the
indices of busy DN.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xkrogen commented on a diff in pull request #4201: HDFS-16547. [SBN read] Namenode in safe mode should not be transfered to observer state
xkrogen commented on code in PR #4201:
URL: https://github.com/apache/hadoop/pull/4201#discussion_r863153584
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -1899,6 +1899,10 @@ synchronized void transitionToStandby() throws
IOException {
synchronized void transitionToObserver() throws IOException {
String operationName = "transitionToObserver";
namesystem.checkSuperuserPrivilege(operationName);
+if (namesystem.isInSafeMode()) {
Review Comment:
I think we can guard this by `notBecomeActiveInSafemode`. Though the config
claims to be about "active" status, the logic in `monitorHealth` just generally
considers a standby NN as unhealthy if it's in safemode, and I think the intent
here is the same as with that config.
Also: `namesystem.isInSafeMode()` -> `isInSafeMode()` (`NameNode` redefines
this method)
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java:
##
@@ -301,6 +304,40 @@ public void testManualFailoverWithDFSHAAdmin() throws
Exception {
waitForHAState(1, HAServiceState.STANDBY);
}
+ /**
+ * Tests that a Namenode in safe mode should not be transfer to observer
state.
+ */
+ @Test
+ public void testManualFailoverWithDFSHAAdminInSafemode() throws Exception {
+startCluster();
+NamenodeProtocols nn1 = cluster.getNameNode(1).getRpcServer();
+
+// Enter safe mode.
+nn1.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_ENTER, false);
+// Test NameNodeRpcServer.
+LambdaTestUtils.intercept(SafeModeException.class,
+"Cannot transition to observer. Name node is in safe mode",
+() -> nn1.transitionToObserver(
+new StateChangeRequestInfo(RequestSource.REQUEST_BY_USER_FORCED)));
+
+// Test DFSHAAdmin.
+DFSHAAdmin tool = new DFSHAAdmin();
+tool.setConf(conf);
+System.setIn(new ByteArrayInputStream("yes\n".getBytes()));
+int result = tool.run(
+new String[]{"-transitionToObserver", "-forcemanual", "nn2"});
+assertEquals("State transition returned: " + result, -1, result);
Review Comment:
This should be in `TestDFSHAAdminMiniCluster`
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java:
##
@@ -301,6 +304,40 @@ public void testManualFailoverWithDFSHAAdmin() throws
Exception {
waitForHAState(1, HAServiceState.STANDBY);
}
+ /**
+ * Tests that a Namenode in safe mode should not be transfer to observer
state.
+ */
+ @Test
+ public void testManualFailoverWithDFSHAAdminInSafemode() throws Exception {
+startCluster();
+NamenodeProtocols nn1 = cluster.getNameNode(1).getRpcServer();
+
+// Enter safe mode.
+nn1.setSafeMode(HdfsConstants.SafeModeAction.SAFEMODE_ENTER, false);
+// Test NameNodeRpcServer.
+LambdaTestUtils.intercept(SafeModeException.class,
+"Cannot transition to observer. Name node is in safe mode",
+() -> nn1.transitionToObserver(
+new StateChangeRequestInfo(RequestSource.REQUEST_BY_USER_FORCED)));
Review Comment:
This should probably be in `TestHASafeMode`, where we already have
`testTransitionToActiveWhenSafeMode`
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -1899,6 +1899,10 @@ synchronized void transitionToStandby() throws
IOException {
synchronized void transitionToObserver() throws IOException {
String operationName = "transitionToObserver";
namesystem.checkSuperuserPrivilege(operationName);
+if (namesystem.isInSafeMode()) {
+ throw namesystem.newSafemodeException("Cannot transition to " +
+ OBSERVER_STATE);
Review Comment:
Consolidate this logic with the exception from `transitionToActive`:
```java
if (notBecomeActiveInSafemode && isInSafeMode()) {
throw new ServiceFailedException(getRole() + " still not leave
safemode");
}
```
I don't think we need to make `newSafemodeException` public, seems fine to
just throw a new exception here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] xkrogen commented on pull request #4209: HDFS-16550. [SBN read] Improper cache-size for journal node may cause cluster crash
xkrogen commented on PR #4209: URL: https://github.com/apache/hadoop/pull/4209#issuecomment-1115302581 I think this change is a bit too restrictive. There may well be valid use cases for setting it above the 90% threshold. For example if you configured a 100GB heap, you really don't need 10GB of non-cache overhead, so you could safely allocate 95GB for the cache. If we want to add fail-fast behavior, I would say it should only apply when `cache size >= heap size`. This is clearly invalid -- you need at least _some_ overhead heap memory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a diff in pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
jojochuang commented on code in PR #4107: URL: https://github.com/apache/hadoop/pull/4107#discussion_r863122472 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java: ## @@ -632,6 +638,20 @@ private static void printDataNodeReports(DistributedFileSystem dfs, } } + private static void printSlowDataNodeReports(DistributedFileSystem dfs, boolean listNodes, Review Comment: Fantastic. Thanks for offering the output screenshot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4256: [DO NOT MERGE] Test YETUS-1060. github status should use htmlreport
hadoop-yetus commented on PR #4256:
URL: https://github.com/apache/hadoop/pull/4256#issuecomment-1115269243
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | shadedclient | 22m 4s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | shadedclient | 19m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 73m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4256/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4256 |
| Optional Tests | dupname asflicense codespell detsecrets shellcheck
shelldocs |
| uname | Linux e128ddd54139 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f757ab439fe0a0cb272c633b17375f125f2ddd05 |
| Max. process+thread count | 720 (vs. ulimit of 5500) |
| modules | C: U: |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4256/1/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on pull request #4252: HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits
jojochuang commented on PR #4252: URL: https://github.com/apache/hadoop/pull/4252#issuecomment-1115254568 @tasanuma @ferhui it would be great to have your review on EC PRs. @umamaheswararao @sodonnel FYI. Please check if Ozone EC is susceptible to the same issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn opened a new pull request, #4257: MAPREDUCE-7376. AggregateWordCount fetches wrong results.
ayushtkn opened a new pull request, #4257: URL: https://github.com/apache/hadoop/pull/4257 ### Description of PR Fixes AggregateWordCount ### How was this patch tested? ``` hadoop-3.4.0-SNAPSHOT % bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.0-SNAPSHOT.jar aggregatewordcount /testData /testOut 1 textinputformat hadoop-3.4.0-SNAPSHOT % bin/hdfs dfs -cat /testOut/part-r-0 Bye 1 Goodbye 1 Hadoop 2 Hello2 World2 ``` ``/testData`` had two files: `wc01.txt:` Hello World Bye World `wc02.txt:` Hello Hadoop Goodbye Hadoop ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on a diff in pull request #4252: HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits
jojochuang commented on code in PR #4252:
URL: https://github.com/apache/hadoop/pull/4252#discussion_r863097191
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestReconstructStripedBlocks.java:
##
@@ -430,4 +433,86 @@ public void testReconstructionWork() throws Exception {
dfsCluster.shutdown();
}
}
+ private byte[] writeStripedFile(DistributedFileSystem fs, Path ecFile,
+ int writeBytes) throws Exception {
+byte[] bytes = StripedFileTestUtil.generateBytes(writeBytes);
+DFSTestUtil.writeFile(fs, ecFile, new String(bytes));
+StripedFileTestUtil.waitBlockGroupsReported(fs, ecFile.toString());
+
+return bytes;
+ }
+ @Test
+ public void testReconstrutionWithBusyBlock1() throws Exception {
+//When the index of busy block is smaller than the missing block
+//[0(busy),1(busy),3,4,5,6,7,8]
+int busyNodeIndex1=0;
Review Comment:
```suggestion
int busyNodeIndex1 = 0;
```
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockECReconstructionCommand.java:
##
@@ -78,28 +78,31 @@ public static class BlockECReconstructionInfo {
private String[] targetStorageIDs;
private StorageType[] targetStorageTypes;
private final byte[] liveBlockIndices;
+private final byte[] excludeReconstructedIndices;
private final ErasureCodingPolicy ecPolicy;
public BlockECReconstructionInfo(ExtendedBlock block,
DatanodeInfo[] sources, DatanodeStorageInfo[] targetDnStorageInfo,
-byte[] liveBlockIndices, ErasureCodingPolicy ecPolicy) {
+byte[] liveBlockIndices, byte[] excludeReconstructedIndices,
ErasureCodingPolicy ecPolicy) {
this(block, sources, DatanodeStorageInfo
.toDatanodeInfos(targetDnStorageInfo), DatanodeStorageInfo
.toStorageIDs(targetDnStorageInfo), DatanodeStorageInfo
- .toStorageTypes(targetDnStorageInfo), liveBlockIndices, ecPolicy);
+ .toStorageTypes(targetDnStorageInfo), liveBlockIndices,
+ excludeReconstructedIndices, ecPolicy);
}
public BlockECReconstructionInfo(ExtendedBlock block,
DatanodeInfo[] sources, DatanodeInfo[] targets,
String[] targetStorageIDs, StorageType[] targetStorageTypes,
-byte[] liveBlockIndices, ErasureCodingPolicy ecPolicy) {
+byte[] liveBlockIndices, byte[] excludeReconstructedIndices,
ErasureCodingPolicy ecPolicy) {
this.block = block;
this.sources = sources;
this.targets = targets;
this.targetStorageIDs = targetStorageIDs;
this.targetStorageTypes = targetStorageTypes;
this.liveBlockIndices = liveBlockIndices == null ?
new byte[]{} : liveBlockIndices;
+ this.excludeReconstructedIndices=excludeReconstructedIndices;
Review Comment:
```suggestion
this.excludeReconstructedIndices = excludeReconstructedIndices;
```
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/erasurecoding.proto:
##
@@ -107,7 +107,8 @@ message BlockECReconstructionInfoProto {
required StorageUuidsProto targetStorageUuids = 4;
required StorageTypesProto targetStorageTypes = 5;
required bytes liveBlockIndices = 6;
- required ErasureCodingPolicyProto ecPolicy = 7;
+ required bytes excludeReconstructedIndices = 7;
Review Comment:
This is going to be incompatible with older versions. Please restore the
order.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedWriter.java:
##
@@ -123,13 +123,14 @@ void init() throws IOException {
private void initTargetIndices() {
BitSet bitset = reconstructor.getLiveBitSet();
+BitSet excludebitset=reconstructor.getExcludeBitSet();
int m = 0;
hasValidTargets = false;
for (int i = 0; i < dataBlkNum + parityBlkNum; i++) {
if (!bitset.get(i)) {
if (reconstructor.getBlockLen(i) > 0) {
- if (m < targets.length) {
+ if (m < targets.length&& !excludebitset.get(i)) {
Review Comment:
check style: please add space between targets.length and &&
```suggestion
if (m < targets.length && !excludebitset.get(i)) {
```
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructionInfo.java:
##
@@ -41,26 +41,28 @@ public class StripedReconstructionInfo {
private final DatanodeInfo[] targets;
private final StorageType[] targetStorageTypes;
private final String[] targetStorageIds;
+ private final byte[] excludeReconstructedIndices;
public StripedReconstructionInfo(ExtendedBlock blockGroup,
ErasureCodingPolicy ecPolicy, byte[] liveIndices, DatanodeInfo[] sources,
byte[] targetIndices) {
this(blockGroup, ecPolicy, liveIndices, sources, targetIndices, null,
-null, null);
+
[jira] [Work logged] (HADOOP-18219) Fix shadedclient test failure
[
https://issues.apache.org/jira/browse/HADOOP-18219?focusedWorklogId=765098=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-765098
]
ASF GitHub Bot logged work on HADOOP-18219:
---
Author: ASF GitHub Bot
Created on: 02/May/22 18:50
Start Date: 02/May/22 18:50
Worklog Time Spent: 10m
Work Description: pjfanning commented on PR #4254:
URL: https://github.com/apache/hadoop/pull/4254#issuecomment-1115242681
Thanks @aajisaka formaking this fix
Issue Time Tracking
---
Worklog Id: (was: 765098)
Time Spent: 2h 10m (was: 2h)
> Fix shadedclient test failure
> -
>
> Key: HADOOP-18219
> URL: https://issues.apache.org/jira/browse/HADOOP-18219
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
> Environment: Debian 10
>Reporter: Gautham Banasandra
>Assignee: Akira Ajisaka
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Two of the shaded client tests are failing on Debian 10 ever since this
> commit -
> https://github.com/apache/hadoop/commit/63187083cc3b9bb1c1e90e692e271958561f9cc8.
> The failures are as follows -
> 1st test failure -
> {code}
> [INFO] Running org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 18.315 s <<< FAILURE! - in org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] useWebHDFS(org.apache.hadoop.example.ITUseMiniCluster) Time elapsed:
> 12.048 s <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:384)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:129)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:500)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:123)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:333)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.example.ITUseMiniCluster.clusterUp(ITUseMiniCluster.java:84)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
>
[GitHub] [hadoop] pjfanning commented on pull request #4254: HADOOP-18219. Fix shadedclient test failure
pjfanning commented on PR #4254: URL: https://github.com/apache/hadoop/pull/4254#issuecomment-1115242681 Thanks @aajisaka formaking this fix -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jteagles commented on pull request #4242: YARN-11116. Migrate Times util from SimpleDateFormat to thread-safe D…
jteagles commented on PR #4242: URL: https://github.com/apache/hadoop/pull/4242#issuecomment-1115230847 @aajisaka, good catch on jdk 7. Thanks so much on this jira. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #4242: YARN-11116. Migrate Times util from SimpleDateFormat to thread-safe D…
aajisaka commented on PR #4242: URL: https://github.com/apache/hadoop/pull/4242#issuecomment-1115223462 @jteagles, I cannot backport this to 2.10.x because 2.10.x still supports JDK 7. > How to update the release branches Just cherry-picking the trunk commit to the branches is fine. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jteagles commented on pull request #4242: YARN-11116. Migrate Times util from SimpleDateFormat to thread-safe D…
jteagles commented on PR #4242: URL: https://github.com/apache/hadoop/pull/4242#issuecomment-1115215449 @aajisaka, I was hoping to backport this to earlier lines. Do you see any problem with me backporting to 2.10.x? Perhaps that is too far back. I was taking a look at https://cwiki.apache.org/confluence/display/HADOOP/GitHub+Integration#GitHubIntegration-MergingaPR(forcommitters) to catch myself up with the updated github process. Before, I would just cherry-pick at this now that it is merged to trunk, but don't see the exact steps to follow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #4242: YARN-11116. Migrate Times util from SimpleDateFormat to thread-safe D…
aajisaka merged PR #4242: URL: https://github.com/apache/hadoop/pull/4242 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #4222: YARN-10187. Removing hadoop-yarn-project/hadoop-yarn/README as it is …
aajisaka commented on PR #4222: URL: https://github.com/apache/hadoop/pull/4222#issuecomment-1115201904 Merged. Thank you @ashutoshcipher -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #4222: YARN-10187. Removing hadoop-yarn-project/hadoop-yarn/README as it is …
aajisaka merged PR #4222: URL: https://github.com/apache/hadoop/pull/4222 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka opened a new pull request, #4256: [DO NOT MERGE] Test YETUS-1060. github status should use htmlreport
aajisaka opened a new pull request, #4256: URL: https://github.com/apache/hadoop/pull/4256 ### Description of PR Testing https://github.com/apache/yetus/pull/261 in Apache Hadoop repo for upcoming Yetus 0.14.0 release. ### How was this patch tested? Not tested locally. ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18069) CVE-2021-0341 in okhttp@2.7.5 detected in hdfs-client
[
https://issues.apache.org/jira/browse/HADOOP-18069?focusedWorklogId=765063=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-765063
]
ASF GitHub Bot logged work on HADOOP-18069:
---
Author: ASF GitHub Bot
Created on: 02/May/22 17:54
Start Date: 02/May/22 17:54
Worklog Time Spent: 10m
Work Description: aajisaka commented on code in PR #4229:
URL: https://github.com/apache/hadoop/pull/4229#discussion_r863056324
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/oauth2/ConfRefreshTokenBasedAccessTokenProvider.java:
##
@@ -102,32 +102,29 @@ public synchronized String getAccessToken() throws
IOException {
}
void refresh() throws IOException {
-try {
- OkHttpClient client = new OkHttpClient();
- client.setConnectTimeout(URLConnectionFactory.DEFAULT_SOCKET_TIMEOUT,
- TimeUnit.MILLISECONDS);
- client.setReadTimeout(URLConnectionFactory.DEFAULT_SOCKET_TIMEOUT,
-TimeUnit.MILLISECONDS);
+OkHttpClient client =
+new
OkHttpClient.Builder().connectTimeout(URLConnectionFactory.DEFAULT_SOCKET_TIMEOUT,
+TimeUnit.MILLISECONDS)
+.readTimeout(URLConnectionFactory.DEFAULT_SOCKET_TIMEOUT,
TimeUnit.MILLISECONDS)
+.build();
- String bodyString = Utils.postBody(GRANT_TYPE, REFRESH_TOKEN,
- REFRESH_TOKEN, refreshToken,
- CLIENT_ID, clientId);
+String bodyString =
+Utils.postBody(GRANT_TYPE, REFRESH_TOKEN, REFRESH_TOKEN, refreshToken,
CLIENT_ID, clientId);
- RequestBody body = RequestBody.create(URLENCODED, bodyString);
-
- Request request = new Request.Builder()
- .url(refreshURL)
- .post(body)
- .build();
- Response responseBody = client.newCall(request).execute();
+RequestBody body = RequestBody.create(bodyString, URLENCODED);
+Request request = new Request.Builder().url(refreshURL).post(body).build();
+try (Response responseBody = client.newCall(request).execute()) {
+ if (!responseBody.isSuccessful()) {
+throw new IOException("Unexpected code " + responseBody);
+ }
if (responseBody.code() != HttpStatus.SC_OK) {
-throw new IllegalArgumentException("Received invalid http response: "
-+ responseBody.code() + ", text = " + responseBody.toString());
+throw new IllegalArgumentException(
+"Received invalid http response: " + responseBody.code() + ", text
= "
++ responseBody.toString());
}
- Map response = JsonSerialization.mapReader().readValue(
- responseBody.body().string());
+ Map response =
JsonSerialization.mapReader().readValue(responseBody.body().string());
Review Comment:
`responseBody.body()` looks duplicated. Maybe we can rename to `response`.
Issue Time Tracking
---
Worklog Id: (was: 765063)
Time Spent: 4h 20m (was: 4h 10m)
> CVE-2021-0341 in okhttp@2.7.5 detected in hdfs-client
> ---
>
> Key: HADOOP-18069
> URL: https://issues.apache.org/jira/browse/HADOOP-18069
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 3.3.1
>Reporter: Eugene Shinn (Truveta)
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Our static vulnerability scanner (Fortify On Demand) detected [NVD -
> CVE-2021-0341
> (nist.gov)|https://nvd.nist.gov/vuln/detail/CVE-2021-0341#VulnChangeHistorySection]
> in our application. We traced the vulnerability to a transitive dependency
> coming from hadoop-hdfs-client, which depends on okhttp@2.7.5
> ([hadoop/pom.xml at trunk · apache/hadoop
> (github.com)|https://github.com/apache/hadoop/blob/trunk/hadoop-project/pom.xml#L137]).
> To resolve this issue, okhttp should be upgraded to 4.9.2+ (ref:
> [CVE-2021-0341 · Issue #6724 · square/okhttp
> (github.com)|https://github.com/square/okhttp/issues/6724]).
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a diff in pull request #4229: HADOOP-18069. okhttp@2.7.5 to 4.9.3
aajisaka commented on code in PR #4229:
URL: https://github.com/apache/hadoop/pull/4229#discussion_r863056324
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/oauth2/ConfRefreshTokenBasedAccessTokenProvider.java:
##
@@ -102,32 +102,29 @@ public synchronized String getAccessToken() throws
IOException {
}
void refresh() throws IOException {
-try {
- OkHttpClient client = new OkHttpClient();
- client.setConnectTimeout(URLConnectionFactory.DEFAULT_SOCKET_TIMEOUT,
- TimeUnit.MILLISECONDS);
- client.setReadTimeout(URLConnectionFactory.DEFAULT_SOCKET_TIMEOUT,
-TimeUnit.MILLISECONDS);
+OkHttpClient client =
+new
OkHttpClient.Builder().connectTimeout(URLConnectionFactory.DEFAULT_SOCKET_TIMEOUT,
+TimeUnit.MILLISECONDS)
+.readTimeout(URLConnectionFactory.DEFAULT_SOCKET_TIMEOUT,
TimeUnit.MILLISECONDS)
+.build();
- String bodyString = Utils.postBody(GRANT_TYPE, REFRESH_TOKEN,
- REFRESH_TOKEN, refreshToken,
- CLIENT_ID, clientId);
+String bodyString =
+Utils.postBody(GRANT_TYPE, REFRESH_TOKEN, REFRESH_TOKEN, refreshToken,
CLIENT_ID, clientId);
- RequestBody body = RequestBody.create(URLENCODED, bodyString);
-
- Request request = new Request.Builder()
- .url(refreshURL)
- .post(body)
- .build();
- Response responseBody = client.newCall(request).execute();
+RequestBody body = RequestBody.create(bodyString, URLENCODED);
+Request request = new Request.Builder().url(refreshURL).post(body).build();
+try (Response responseBody = client.newCall(request).execute()) {
+ if (!responseBody.isSuccessful()) {
+throw new IOException("Unexpected code " + responseBody);
+ }
if (responseBody.code() != HttpStatus.SC_OK) {
-throw new IllegalArgumentException("Received invalid http response: "
-+ responseBody.code() + ", text = " + responseBody.toString());
+throw new IllegalArgumentException(
+"Received invalid http response: " + responseBody.code() + ", text
= "
++ responseBody.toString());
}
- Map response = JsonSerialization.mapReader().readValue(
- responseBody.body().string());
+ Map response =
JsonSerialization.mapReader().readValue(responseBody.body().string());
Review Comment:
`responseBody.body()` looks duplicated. Maybe we can rename to `response`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18069) CVE-2021-0341 in okhttp@2.7.5 detected in hdfs-client
[ https://issues.apache.org/jira/browse/HADOOP-18069?focusedWorklogId=765062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-765062 ] ASF GitHub Bot logged work on HADOOP-18069: --- Author: ASF GitHub Bot Created on: 02/May/22 17:51 Start Date: 02/May/22 17:51 Worklog Time Spent: 10m Work Description: aajisaka commented on PR #4229: URL: https://github.com/apache/hadoop/pull/4229#issuecomment-1115177416 > spotbugs still thinks there is a problem. what is it that it is warning about? > I already used try-with-resources for OkHttpClient and added request checks as well. The warning is about NPE case. Reading the source code, `responseBody.body()` is always non-null. However, spotbugs thinks `responseBody.body()` may be null and it warns "need null check". I think it is because Spotbugs failed to analyze the okhttp classes from Kotlin. I think we can simply ignore the warnings. @ashutoshcipher would you add some entries to ignore the spotbugs warning in `hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml`? Issue Time Tracking --- Worklog Id: (was: 765062) Time Spent: 4h 10m (was: 4h) > CVE-2021-0341 in okhttp@2.7.5 detected in hdfs-client > --- > > Key: HADOOP-18069 > URL: https://issues.apache.org/jira/browse/HADOOP-18069 > Project: Hadoop Common > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.3.1 >Reporter: Eugene Shinn (Truveta) >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Time Spent: 4h 10m > Remaining Estimate: 0h > > Our static vulnerability scanner (Fortify On Demand) detected [NVD - > CVE-2021-0341 > (nist.gov)|https://nvd.nist.gov/vuln/detail/CVE-2021-0341#VulnChangeHistorySection] > in our application. We traced the vulnerability to a transitive dependency > coming from hadoop-hdfs-client, which depends on okhttp@2.7.5 > ([hadoop/pom.xml at trunk · apache/hadoop > (github.com)|https://github.com/apache/hadoop/blob/trunk/hadoop-project/pom.xml#L137]). > To resolve this issue, okhttp should be upgraded to 4.9.2+ (ref: > [CVE-2021-0341 · Issue #6724 · square/okhttp > (github.com)|https://github.com/square/okhttp/issues/6724]). -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #4229: HADOOP-18069. okhttp@2.7.5 to 4.9.3
aajisaka commented on PR #4229: URL: https://github.com/apache/hadoop/pull/4229#issuecomment-1115177416 > spotbugs still thinks there is a problem. what is it that it is warning about? > I already used try-with-resources for OkHttpClient and added request checks as well. The warning is about NPE case. Reading the source code, `responseBody.body()` is always non-null. However, spotbugs thinks `responseBody.body()` may be null and it warns "need null check". I think it is because Spotbugs failed to analyze the okhttp classes from Kotlin. I think we can simply ignore the warnings. @ashutoshcipher would you add some entries to ignore the spotbugs warning in `hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a diff in pull request #4244: YARN-11119. Backport YARN-10538 to branch-2.10
aajisaka commented on code in PR #4244:
URL: https://github.com/apache/hadoop/pull/4244#discussion_r863034596
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCNodeUpdates.java:
##
@@ -98,6 +111,47 @@ public AllocateResponse run() throws Exception {
});
}
+ @Test
+ public void testAMRMRecommissioningNodes() throws Exception {
+MockNM nm1 = rm.registerNode("127.0.0.1:1234", 1);
+MockNM nm2 = rm.registerNode("127.0.0.2:1234", 1);
+rm.drainEvents();
+
+RMApp app1 = rm.submitApp(2000);
+
+// Trigger the scheduling so the AM gets 'launched' on nm1
+nm1.nodeHeartbeat(true);
+
+RMAppAttempt attempt1 = app1.getCurrentAppAttempt();
+MockAM am1 = rm.sendAMLaunched(attempt1.getAppAttemptId());
+
+// register AM returns no unusable node
+am1.registerAppAttempt();
+
+// DECOMMISSION nm2
+syncNodeGracefulDecommission(nm2);
+
+AllocateRequest allocateRequest1 =
+AllocateRequest.newInstance(0, 0F, null, null, null);
+AllocateResponse response1 =
+allocate(attempt1.getAppAttemptId(), allocateRequest1);
+List updatedNodes = response1.getUpdatedNodes();
+Assert.assertEquals(0, updatedNodes.size());
Review Comment:
I noticed the size of `updatedNodes` is 1 in #2564. It is because YARN-6483
is not in branch-2.10.
Now I'm thinking we need to backport YARN-6483 before YARN-10538.
@ashutoshcipher what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18219) Fix shadedclient test failure
[
https://issues.apache.org/jira/browse/HADOOP-18219?focusedWorklogId=765050=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-765050
]
ASF GitHub Bot logged work on HADOOP-18219:
---
Author: ASF GitHub Bot
Created on: 02/May/22 17:07
Start Date: 02/May/22 17:07
Worklog Time Spent: 10m
Work Description: aajisaka commented on PR #4254:
URL: https://github.com/apache/hadoop/pull/4254#issuecomment-1115131143
Thank you @GauthamBanasandra @ashutoshcipher @steveloughran for your reviews.
Issue Time Tracking
---
Worklog Id: (was: 765050)
Time Spent: 2h (was: 1h 50m)
> Fix shadedclient test failure
> -
>
> Key: HADOOP-18219
> URL: https://issues.apache.org/jira/browse/HADOOP-18219
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
> Environment: Debian 10
>Reporter: Gautham Banasandra
>Assignee: Akira Ajisaka
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Two of the shaded client tests are failing on Debian 10 ever since this
> commit -
> https://github.com/apache/hadoop/commit/63187083cc3b9bb1c1e90e692e271958561f9cc8.
> The failures are as follows -
> 1st test failure -
> {code}
> [INFO] Running org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 18.315 s <<< FAILURE! - in org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] useWebHDFS(org.apache.hadoop.example.ITUseMiniCluster) Time elapsed:
> 12.048 s <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:384)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:129)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:500)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:123)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:333)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.example.ITUseMiniCluster.clusterUp(ITUseMiniCluster.java:84)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
>
[GitHub] [hadoop] aajisaka commented on pull request #4254: HADOOP-18219. Fix shadedclient test failure
aajisaka commented on PR #4254: URL: https://github.com/apache/hadoop/pull/4254#issuecomment-1115131143 Thank you @GauthamBanasandra @ashutoshcipher @steveloughran for your reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18219) Fix shadedclient test failure
[
https://issues.apache.org/jira/browse/HADOOP-18219?focusedWorklogId=765033=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-765033
]
ASF GitHub Bot logged work on HADOOP-18219:
---
Author: ASF GitHub Bot
Created on: 02/May/22 16:42
Start Date: 02/May/22 16:42
Worklog Time Spent: 10m
Work Description: GauthamBanasandra closed pull request #4253:
HADOOP-18219. Revert "HADOOP-15983. Use jersey-json that is built to use
jackson2 (#3988)"
URL: https://github.com/apache/hadoop/pull/4253
Issue Time Tracking
---
Worklog Id: (was: 765033)
Time Spent: 1h 50m (was: 1h 40m)
> Fix shadedclient test failure
> -
>
> Key: HADOOP-18219
> URL: https://issues.apache.org/jira/browse/HADOOP-18219
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
> Environment: Debian 10
>Reporter: Gautham Banasandra
>Assignee: Akira Ajisaka
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Two of the shaded client tests are failing on Debian 10 ever since this
> commit -
> https://github.com/apache/hadoop/commit/63187083cc3b9bb1c1e90e692e271958561f9cc8.
> The failures are as follows -
> 1st test failure -
> {code}
> [INFO] Running org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 18.315 s <<< FAILURE! - in org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] useWebHDFS(org.apache.hadoop.example.ITUseMiniCluster) Time elapsed:
> 12.048 s <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:384)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:129)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:500)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:123)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:333)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.example.ITUseMiniCluster.clusterUp(ITUseMiniCluster.java:84)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
>
[jira] [Work logged] (HADOOP-18219) Fix shadedclient test failure
[
https://issues.apache.org/jira/browse/HADOOP-18219?focusedWorklogId=765032=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-765032
]
ASF GitHub Bot logged work on HADOOP-18219:
---
Author: ASF GitHub Bot
Created on: 02/May/22 16:42
Start Date: 02/May/22 16:42
Worklog Time Spent: 10m
Work Description: GauthamBanasandra commented on PR #4253:
URL: https://github.com/apache/hadoop/pull/4253#issuecomment-1115108791
Abandoning this PR since the issue was fixed by
https://github.com/apache/hadoop/pull/4254.
Issue Time Tracking
---
Worklog Id: (was: 765032)
Time Spent: 1h 40m (was: 1.5h)
> Fix shadedclient test failure
> -
>
> Key: HADOOP-18219
> URL: https://issues.apache.org/jira/browse/HADOOP-18219
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
> Environment: Debian 10
>Reporter: Gautham Banasandra
>Assignee: Akira Ajisaka
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Two of the shaded client tests are failing on Debian 10 ever since this
> commit -
> https://github.com/apache/hadoop/commit/63187083cc3b9bb1c1e90e692e271958561f9cc8.
> The failures are as follows -
> 1st test failure -
> {code}
> [INFO] Running org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 18.315 s <<< FAILURE! - in org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] useWebHDFS(org.apache.hadoop.example.ITUseMiniCluster) Time elapsed:
> 12.048 s <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:384)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:129)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:500)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:123)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:333)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.example.ITUseMiniCluster.clusterUp(ITUseMiniCluster.java:84)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
>
[GitHub] [hadoop] GauthamBanasandra closed pull request #4253: HADOOP-18219. Revert "HADOOP-15983. Use jersey-json that is built to use jackson2 (#3988)"
GauthamBanasandra closed pull request #4253: HADOOP-18219. Revert "HADOOP-15983. Use jersey-json that is built to use jackson2 (#3988)" URL: https://github.com/apache/hadoop/pull/4253 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra commented on pull request #4253: HADOOP-18219. Revert "HADOOP-15983. Use jersey-json that is built to use jackson2 (#3988)"
GauthamBanasandra commented on PR #4253: URL: https://github.com/apache/hadoop/pull/4253#issuecomment-1115108791 Abandoning this PR since the issue was fixed by https://github.com/apache/hadoop/pull/4254. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18167) Add metrics to track delegation token secret manager operations
[
https://issues.apache.org/jira/browse/HADOOP-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530789#comment-17530789
]
Ayush Saxena commented on HADOOP-18167:
---
Not sure though but seems this is creating some problem for me while starting
the ResourceManager:
{noformat}
Caused by: java.io.IOException: Problem starting http server
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1389)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:475)
... 4 more
Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source
DelegationTokenSecretManagerMetrics already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
at
org.apache.hadoop.metrics2.MetricsSystem.register(MetricsSystem.java:71)
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$DelegationTokenSecretManagerMetrics.create(AbstractDelegationTokenSecretManager.java:878)
at
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.(AbstractDelegationTokenSecretManager.java:152)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$DelegationTokenSecretManager.(DelegationTokenManager.java:72)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.(DelegationTokenManager.java:122)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.initTokenManager(DelegationTokenAuthenticationHandler.java:161)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.init(DelegationTokenAuthenticationHandler.java:130)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeAuthHandler(AuthenticationFilter.java:194)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.initializeAuthHandler(DelegationTokenAuthenticationFilter.java:214)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:180)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:180)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.init(RMAuthenticationFilter.java:53)
{noformat}
Is something need to configured, or by any chance this metrics is being
registered twice in RM?
> Add metrics to track delegation token secret manager operations
> ---
>
> Key: HADOOP-18167
> URL: https://issues.apache.org/jira/browse/HADOOP-18167
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Hector Sandoval Chaverri
>Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-18167-branch-2.10-2.patch,
> HADOOP-18167-branch-2.10-3.patch, HADOOP-18167-branch-2.10-4.patch,
> HADOOP-18167-branch-2.10.patch, HADOOP-18167-branch-3.3.patch
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> New metrics to track operations that store, update and remove delegation
> tokens in implementations of AbstractDelegationTokenSecretManager. This will
> help evaluate the impact of using different secret managers and add
> optimizations.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18219) Fix shadedclient test failure
[
https://issues.apache.org/jira/browse/HADOOP-18219?focusedWorklogId=764986=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-764986
]
ASF GitHub Bot logged work on HADOOP-18219:
---
Author: ASF GitHub Bot
Created on: 02/May/22 15:18
Start Date: 02/May/22 15:18
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4253:
URL: https://github.com/apache/hadoop/pull/4253#issuecomment-1115024879
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 45m 24s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | shellcheck | 0m 1s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 39s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 39s | | trunk passed |
| +1 :green_heart: | compile | 21m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 20m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 117m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 44s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 38m 40s | | the patch passed |
| +1 :green_heart: | compile | 21m 3s | | the patch passed |
| -1 :x: | cc | 21m 3s |
[/results-compile-cc-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4253/1/artifact/out/results-compile-cc-root.txt)
| root generated 6 new + 199 unchanged - 6 fixed = 205 total (was 205) |
| +1 :green_heart: | golang | 21m 3s | | the patch passed |
| +1 :green_heart: | javac | 21m 3s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 19m 55s | | the patch passed |
| +1 :green_heart: | xml | 0m 15s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 8m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 1014m 36s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4253/1/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 9s | | The patch does not
generate ASF License warnings. |
| | | 1294m 12s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.security.TestRouterSecurityManager |
| | hadoop.yarn.sls.TestSLSRunner |
| | hadoop.yarn.sls.TestSLSStreamAMSynth |
| | hadoop.yarn.sls.TestSLSDagAMSimulator |
| | hadoop.yarn.sls.nodemanager.TestNMSimulator |
| | hadoop.yarn.sls.TestSLSGenericSynth |
| | hadoop.yarn.sls.TestReservationSystemInvariants |
| | hadoop.yarn.sls.appmaster.TestAMSimulator |
| | hadoop.yarn.client.TestGetGroups |
| |
hadoop.yarn.client.TestResourceManagerAdministrationProtocolPBClientImpl |
| | hadoop.yarn.server.router.webapp.TestRouterWebServicesREST |
| | hadoop.yarn.server.timeline.webapp.TestTimelineWebServicesWithSSL |
| |
hadoop.yarn.server.timeline.security.TestTimelineAuthenticationFilterForV1 |
| |
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
|
| | hadoop.yarn.server.resourcemanager.TestRMHA |
| | hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher |
| |
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication |
| | hadoop.yarn.webapp.TestRMWithXFSFilter |
| | hadoop.yarn.server.resourcemanager.TestClientRMService |
| | hadoop.mapred.TestLocalDistributedCacheManager |
| | hadoop.crypto.TestCryptoCodec |
| | hadoop.service.launcher.TestServiceInterruptHandling |
| | hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4253: HADOOP-18219. Revert "HADOOP-15983. Use jersey-json that is built to use jackson2 (#3988)"
hadoop-yetus commented on PR #4253:
URL: https://github.com/apache/hadoop/pull/4253#issuecomment-1115024879
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 45m 24s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | shellcheck | 0m 1s | | Shellcheck was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 39s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 39s | | trunk passed |
| +1 :green_heart: | compile | 21m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 20m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 53s | | trunk passed |
| +1 :green_heart: | shadedclient | 117m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 44s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 38m 40s | | the patch passed |
| +1 :green_heart: | compile | 21m 3s | | the patch passed |
| -1 :x: | cc | 21m 3s |
[/results-compile-cc-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4253/1/artifact/out/results-compile-cc-root.txt)
| root generated 6 new + 199 unchanged - 6 fixed = 205 total (was 205) |
| +1 :green_heart: | golang | 21m 3s | | the patch passed |
| +1 :green_heart: | javac | 21m 3s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 19m 55s | | the patch passed |
| +1 :green_heart: | xml | 0m 15s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 8m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 1014m 36s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4253/1/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 9s | | The patch does not
generate ASF License warnings. |
| | | 1294m 12s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.security.TestRouterSecurityManager |
| | hadoop.yarn.sls.TestSLSRunner |
| | hadoop.yarn.sls.TestSLSStreamAMSynth |
| | hadoop.yarn.sls.TestSLSDagAMSimulator |
| | hadoop.yarn.sls.nodemanager.TestNMSimulator |
| | hadoop.yarn.sls.TestSLSGenericSynth |
| | hadoop.yarn.sls.TestReservationSystemInvariants |
| | hadoop.yarn.sls.appmaster.TestAMSimulator |
| | hadoop.yarn.client.TestGetGroups |
| |
hadoop.yarn.client.TestResourceManagerAdministrationProtocolPBClientImpl |
| | hadoop.yarn.server.router.webapp.TestRouterWebServicesREST |
| | hadoop.yarn.server.timeline.webapp.TestTimelineWebServicesWithSSL |
| |
hadoop.yarn.server.timeline.security.TestTimelineAuthenticationFilterForV1 |
| |
hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
|
| | hadoop.yarn.server.resourcemanager.TestRMHA |
| | hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher |
| |
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication |
| | hadoop.yarn.webapp.TestRMWithXFSFilter |
| | hadoop.yarn.server.resourcemanager.TestClientRMService |
| | hadoop.mapred.TestLocalDistributedCacheManager |
| | hadoop.crypto.TestCryptoCodec |
| | hadoop.service.launcher.TestServiceInterruptHandling |
| | hadoop.crypto.TestCryptoStreamsWithOpensslSm4CtrCryptoCodec |
| | hadoop.crypto.key.kms.server.TestKMSWithZK |
| | hadoop.crypto.key.kms.server.TestKMS |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4253/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4253 |
| Optional Tests | dupname asflicense
[GitHub] [hadoop] slfan1989 commented on pull request #4250: YARN-11121:Conditional judgment Add in getClusterMetrics
slfan1989 commented on PR #4250: URL: https://github.com/apache/hadoop/pull/4250#issuecomment-1114894552 When the PR was automatically checked, it was found that the test could not pass, and the following error was reported in the local test. ``` Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source DelegationTokenSecretManagerMetrics already exists! at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152) at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229) at org.apache.hadoop.metrics2.MetricsSystem.register(MetricsSystem.java:71) at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$DelegationTokenSecretManagerMetrics.create(AbstractDelegationTokenSecretManager.java:878) at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.(AbstractDelegationTokenSecretManager.java:152) at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$DelegationTokenSecretManager.(DelegationTokenManager.java:72) at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.(DelegationTokenManager.java:122) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.initTokenManager(DelegationTokenAuthenticationHandler.java:161) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.init(DelegationTokenAuthenticationHandler.java:130) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeAuthHandler(AuthenticationFilter.java:194) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.initializeAuthHandler(DelegationTokenAuthenticationFilter.java:214) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:180) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:180) at org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.init(RMAuthenticationFilter.java:53) at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:140) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18217) shutdownhookmanager should not be multithreaded (deadlock possible)
[
https://issues.apache.org/jira/browse/HADOOP-18217?focusedWorklogId=764920=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-764920
]
ASF GitHub Bot logged work on HADOOP-18217:
---
Author: ASF GitHub Bot
Created on: 02/May/22 12:47
Start Date: 02/May/22 12:47
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4255:
URL: https://github.com/apache/hadoop/pull/4255#issuecomment-1114806690
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 1s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 26s | | trunk passed |
| +1 :green_heart: | compile | 25m 9s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 50s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 7s | | trunk passed |
| -1 :x: | javadoc | 1m 38s |
[/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4255/1/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| hadoop-common in trunk failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| +1 :green_heart: | javadoc | 2m 9s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 24m 2s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 2s | | the patch passed |
| +1 :green_heart: | compile | 21m 36s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 23s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4255/1/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 3 new + 2
unchanged - 0 fixed = 5 total (was 2) |
| +1 :green_heart: | mvnsite | 1m 55s | | the patch passed |
| -1 :x: | javadoc | 1m 26s |
[/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4255/1/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| hadoop-common in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| +1 :green_heart: | javadoc | 1m 59s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 8s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 13s | | The patch does not
generate ASF License warnings. |
| | | 246m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4255/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4255 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs
[GitHub] [hadoop] hadoop-yetus commented on pull request #4255: HADOOP-18217. ExitUtil synchronized blocks reduced to avoid exit bloc…
hadoop-yetus commented on PR #4255:
URL: https://github.com/apache/hadoop/pull/4255#issuecomment-1114806690
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 1s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 26s | | trunk passed |
| +1 :green_heart: | compile | 25m 9s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 50s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 7s | | trunk passed |
| -1 :x: | javadoc | 1m 38s |
[/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4255/1/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| hadoop-common in trunk failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| +1 :green_heart: | javadoc | 2m 9s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 24m 2s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 2s | | the patch passed |
| +1 :green_heart: | compile | 21m 36s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 23s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4255/1/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 3 new + 2
unchanged - 0 fixed = 5 total (was 2) |
| +1 :green_heart: | mvnsite | 1m 55s | | the patch passed |
| -1 :x: | javadoc | 1m 26s |
[/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4255/1/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| hadoop-common in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| +1 :green_heart: | javadoc | 1m 59s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 8s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 13s | | The patch does not
generate ASF License warnings. |
| | | 246m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4255/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4255 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0b98321b98d7 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 55b71c8b9bf2840f3bae05c97703eb2c08672b58 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4235: YARN-11114. RMWebServices returns only apps matching exactly the submitted queue name
hadoop-yetus commented on PR #4235:
URL: https://github.com/apache/hadoop/pull/4235#issuecomment-1114764977
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 36s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 15s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 0s | | the patch passed |
| +1 :green_heart: | compile | 1m 5s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 1m 5s | | the patch passed |
| +1 :green_heart: | compile | 0m 58s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 58s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4235/5/artifact/out/blanks-eol.txt)
| The patch has 3 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 0m 48s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4235/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 4 new + 44 unchanged - 0 fixed = 48 total (was 44) |
| +1 :green_heart: | mvnsite | 1m 0s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 96m 50s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4235/5/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 197m 32s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps |
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
|
| | hadoop.yarn.server.resourcemanager.TestRMHA |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebappAuthentication |
| | hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher |
| |
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher |
| | hadoop.yarn.webapp.TestRMWithXFSFilter |
| | hadoop.yarn.server.resourcemanager.TestClientRMService |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4235/5/artifact/out/Dockerfile
|
| GITHUB PR |
[jira] [Updated] (HADOOP-15983) Use jersey-json that is built to use jackson2
[ https://issues.apache.org/jira/browse/HADOOP-15983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15983: Release Note: Use modified jersey-json 1.20 in https://github.com/pjfanning/jersey-1.x/tree/v1.20 that uses Jackson 2.x. By this change, Jackson 1.x dependency has been removed from Hadoop. downstream applications which explicitly exclude jersey from transitive dependencies must now exclude com.github.pjfanning:jersey-json was:Use modified jersey-json 1.20 in https://github.com/pjfanning/jersey-1.x/tree/v1.20 that uses Jackson 2.x. By this change, Jackson 1.x dependency has been removed from Hadoop. > Use jersey-json that is built to use jackson2 > - > > Key: HADOOP-15983 > URL: https://issues.apache.org/jira/browse/HADOOP-15983 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > moves to a fork of jersey 1 which removes the jackson 1 dependency. > when cherrypicking this, HADOOP-18219 MUST also be included -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15983) Use jersey-json that is built to use jackson2
[ https://issues.apache.org/jira/browse/HADOOP-15983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15983: Description: moves to a fork of jersey 1 which removes the jackson 1 dependency. when cherrypicking this, HADOOP-18219 MUST also be included > Use jersey-json that is built to use jackson2 > - > > Key: HADOOP-15983 > URL: https://issues.apache.org/jira/browse/HADOOP-15983 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > moves to a fork of jersey 1 which removes the jackson 1 dependency. > when cherrypicking this, HADOOP-18219 MUST also be included -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18219) Fix shadedclient test failure
[
https://issues.apache.org/jira/browse/HADOOP-18219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-18219.
-
Resolution: Fixed
merged to trunk
> Fix shadedclient test failure
> -
>
> Key: HADOOP-18219
> URL: https://issues.apache.org/jira/browse/HADOOP-18219
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
> Environment: Debian 10
>Reporter: Gautham Banasandra
>Assignee: Akira Ajisaka
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Two of the shaded client tests are failing on Debian 10 ever since this
> commit -
> https://github.com/apache/hadoop/commit/63187083cc3b9bb1c1e90e692e271958561f9cc8.
> The failures are as follows -
> 1st test failure -
> {code}
> [INFO] Running org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 18.315 s <<< FAILURE! - in org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] useWebHDFS(org.apache.hadoop.example.ITUseMiniCluster) Time elapsed:
> 12.048 s <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:384)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:129)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:500)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:123)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:333)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.example.ITUseMiniCluster.clusterUp(ITUseMiniCluster.java:84)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by:
[jira] [Assigned] (HADOOP-18219) Fix shadedclient test failure
[
https://issues.apache.org/jira/browse/HADOOP-18219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reassigned HADOOP-18219:
---
Assignee: Akira Ajisaka (was: PJ Fanning)
> Fix shadedclient test failure
> -
>
> Key: HADOOP-18219
> URL: https://issues.apache.org/jira/browse/HADOOP-18219
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
> Environment: Debian 10
>Reporter: Gautham Banasandra
>Assignee: Akira Ajisaka
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Two of the shaded client tests are failing on Debian 10 ever since this
> commit -
> https://github.com/apache/hadoop/commit/63187083cc3b9bb1c1e90e692e271958561f9cc8.
> The failures are as follows -
> 1st test failure -
> {code}
> [INFO] Running org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 18.315 s <<< FAILURE! - in org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] useWebHDFS(org.apache.hadoop.example.ITUseMiniCluster) Time elapsed:
> 12.048 s <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:384)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:129)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:500)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:123)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:333)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.example.ITUseMiniCluster.clusterUp(ITUseMiniCluster.java:84)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by:
[jira] [Updated] (HADOOP-18219) Fix shadedclient test failure
[
https://issues.apache.org/jira/browse/HADOOP-18219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18219:
Fix Version/s: 3.4.0
> Fix shadedclient test failure
> -
>
> Key: HADOOP-18219
> URL: https://issues.apache.org/jira/browse/HADOOP-18219
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
> Environment: Debian 10
>Reporter: Gautham Banasandra
>Assignee: Akira Ajisaka
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Two of the shaded client tests are failing on Debian 10 ever since this
> commit -
> https://github.com/apache/hadoop/commit/63187083cc3b9bb1c1e90e692e271958561f9cc8.
> The failures are as follows -
> 1st test failure -
> {code}
> [INFO] Running org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 18.315 s <<< FAILURE! - in org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] useWebHDFS(org.apache.hadoop.example.ITUseMiniCluster) Time elapsed:
> 12.048 s <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:384)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:129)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:500)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:123)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:333)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.example.ITUseMiniCluster.clusterUp(ITUseMiniCluster.java:84)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by:
[jira] [Work logged] (HADOOP-18219) Fix shadedclient test failure
[
https://issues.apache.org/jira/browse/HADOOP-18219?focusedWorklogId=764900=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-764900
]
ASF GitHub Bot logged work on HADOOP-18219:
---
Author: ASF GitHub Bot
Created on: 02/May/22 11:25
Start Date: 02/May/22 11:25
Worklog Time Spent: 10m
Work Description: steveloughran merged PR #4254:
URL: https://github.com/apache/hadoop/pull/4254
Issue Time Tracking
---
Worklog Id: (was: 764900)
Time Spent: 1h 20m (was: 1h 10m)
> Fix shadedclient test failure
> -
>
> Key: HADOOP-18219
> URL: https://issues.apache.org/jira/browse/HADOOP-18219
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
> Environment: Debian 10
>Reporter: Gautham Banasandra
>Assignee: PJ Fanning
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Two of the shaded client tests are failing on Debian 10 ever since this
> commit -
> https://github.com/apache/hadoop/commit/63187083cc3b9bb1c1e90e692e271958561f9cc8.
> The failures are as follows -
> 1st test failure -
> {code}
> [INFO] Running org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 18.315 s <<< FAILURE! - in org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] useWebHDFS(org.apache.hadoop.example.ITUseMiniCluster) Time elapsed:
> 12.048 s <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:384)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:129)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:500)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:123)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:333)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.example.ITUseMiniCluster.clusterUp(ITUseMiniCluster.java:84)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
>
[GitHub] [hadoop] steveloughran merged pull request #4254: HADOOP-18219. Fix shadedclient test failure
steveloughran merged PR #4254: URL: https://github.com/apache/hadoop/pull/4254 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18217) shutdownhookmanager should not be multithreaded (deadlock possible)
[
https://issues.apache.org/jira/browse/HADOOP-18217?focusedWorklogId=764899=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-764899
]
ASF GitHub Bot logged work on HADOOP-18217:
---
Author: ASF GitHub Bot
Created on: 02/May/22 11:20
Start Date: 02/May/22 11:20
Worklog Time Spent: 10m
Work Description: steveloughran commented on code in PR #4255:
URL: https://github.com/apache/hadoop/pull/4255#discussion_r862765210
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java:
##
@@ -226,25 +251,48 @@ public static synchronized void terminate(ExitException
ee)
* trace.
* @throws HaltException if {@link Runtime#halt(int)} is disabled.
*/
- public static synchronized void halt(HaltException ee) throws HaltException {
-int status = ee.getExitCode();
-String msg = ee.getMessage();
-try {
- if (status != 0) {
+ public static void halt(HaltException ee) throws HaltException {
+final int status = ee.getExitCode();
+Error catched = null;
+if (status != 0) {
+ try {
//exit indicates a problem, log it
+String msg = ee.getMessage();
LOG.info("Halt with status {}: {}", status, msg, ee);
+ } catch (Error e) {
+catched = e; // errors have higher priority than HaltException, it may
be re-thrown. OOM and ThreadDeath are 2 examples of Errors to re-throw
+ } catch (Throwable t) {
+// all other kind of throwables are supressed
+ee.addSuppressed(t);
}
-} catch (Exception ignored) {
- // ignore exceptions here, as it may be due to an out of memory situation
}
-if (systemHaltDisabled) {
- LOG.error("Halt called", ee);
- if (!haltCalled()) {
-firstHaltException = ee;
+if (systemHaltDisabled) { // this is a volatile so reading it does not
need a synchronized block
Review Comment:
do think we should move to atomic boolean? it's a bit less efficient, but
this isn't a performance codepath, and it might make for safer code for future
maintenance. there are details about volatile that nobody ever remembers/knows
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java:
##
@@ -200,23 +200,48 @@ public static void resetFirstHaltException() {
* or, if system exits are disabled, rethrow the exception.
* @param ee exit exception
*/
- public static synchronized void terminate(ExitException ee)
+ public static void terminate(ExitException ee)
throws ExitException {
-int status = ee.getExitCode();
-String msg = ee.getMessage();
+final int status = ee.getExitCode();
+Error catched = null;
Review Comment:
`caught`
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java:
##
@@ -200,23 +200,48 @@ public static void resetFirstHaltException() {
* or, if system exits are disabled, rethrow the exception.
* @param ee exit exception
*/
- public static synchronized void terminate(ExitException ee)
+ public static void terminate(ExitException ee)
throws ExitException {
-int status = ee.getExitCode();
-String msg = ee.getMessage();
+final int status = ee.getExitCode();
+Error catched = null;
if (status != 0) {
- //exit indicates a problem, log it
- LOG.debug("Exiting with status {}: {}", status, msg, ee);
- LOG.info("Exiting with status {}: {}", status, msg);
+ try {
+//exit indicates a problem, log it
+String msg = ee.getMessage();
+LOG.debug("Exiting with status {}: {}", status, msg, ee);
+LOG.info("Exiting with status {}: {}", status, msg);
+ } catch (Error e) {
+catched = e; // errors have higher priority than HaltException, it may
be re-thrown. OOM and ThreadDeath are 2 examples of Errors to re-throw
Review Comment:
nit. put comment on line above
Issue Time Tracking
---
Worklog Id: (was: 764899)
Time Spent: 20m (was: 10m)
> shutdownhookmanager should not be multithreaded (deadlock possible)
> ---
>
> Key: HADOOP-18217
> URL: https://issues.apache.org/jira/browse/HADOOP-18217
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 2.10.1
> Environment: linux, windows, any version
>Reporter: Catherinot Remi
>Priority: Minor
> Labels: pull-request-available
> Attachments: wtf.java
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> the ShutdownHookManager class uses an executor to run hooks to have a
> "timeout" notion around them. It does this using a single threaded executor.
> It can leads to deadlock leaving a
[GitHub] [hadoop] steveloughran commented on a diff in pull request #4255: HADOOP-18217. ExitUtil synchronized blocks reduced to avoid exit bloc…
steveloughran commented on code in PR #4255:
URL: https://github.com/apache/hadoop/pull/4255#discussion_r862765210
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java:
##
@@ -226,25 +251,48 @@ public static synchronized void terminate(ExitException
ee)
* trace.
* @throws HaltException if {@link Runtime#halt(int)} is disabled.
*/
- public static synchronized void halt(HaltException ee) throws HaltException {
-int status = ee.getExitCode();
-String msg = ee.getMessage();
-try {
- if (status != 0) {
+ public static void halt(HaltException ee) throws HaltException {
+final int status = ee.getExitCode();
+Error catched = null;
+if (status != 0) {
+ try {
//exit indicates a problem, log it
+String msg = ee.getMessage();
LOG.info("Halt with status {}: {}", status, msg, ee);
+ } catch (Error e) {
+catched = e; // errors have higher priority than HaltException, it may
be re-thrown. OOM and ThreadDeath are 2 examples of Errors to re-throw
+ } catch (Throwable t) {
+// all other kind of throwables are supressed
+ee.addSuppressed(t);
}
-} catch (Exception ignored) {
- // ignore exceptions here, as it may be due to an out of memory situation
}
-if (systemHaltDisabled) {
- LOG.error("Halt called", ee);
- if (!haltCalled()) {
-firstHaltException = ee;
+if (systemHaltDisabled) { // this is a volatile so reading it does not
need a synchronized block
Review Comment:
do think we should move to atomic boolean? it's a bit less efficient, but
this isn't a performance codepath, and it might make for safer code for future
maintenance. there are details about volatile that nobody ever remembers/knows
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java:
##
@@ -200,23 +200,48 @@ public static void resetFirstHaltException() {
* or, if system exits are disabled, rethrow the exception.
* @param ee exit exception
*/
- public static synchronized void terminate(ExitException ee)
+ public static void terminate(ExitException ee)
throws ExitException {
-int status = ee.getExitCode();
-String msg = ee.getMessage();
+final int status = ee.getExitCode();
+Error catched = null;
Review Comment:
`caught`
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java:
##
@@ -200,23 +200,48 @@ public static void resetFirstHaltException() {
* or, if system exits are disabled, rethrow the exception.
* @param ee exit exception
*/
- public static synchronized void terminate(ExitException ee)
+ public static void terminate(ExitException ee)
throws ExitException {
-int status = ee.getExitCode();
-String msg = ee.getMessage();
+final int status = ee.getExitCode();
+Error catched = null;
if (status != 0) {
- //exit indicates a problem, log it
- LOG.debug("Exiting with status {}: {}", status, msg, ee);
- LOG.info("Exiting with status {}: {}", status, msg);
+ try {
+//exit indicates a problem, log it
+String msg = ee.getMessage();
+LOG.debug("Exiting with status {}: {}", status, msg, ee);
+LOG.info("Exiting with status {}: {}", status, msg);
+ } catch (Error e) {
+catched = e; // errors have higher priority than HaltException, it may
be re-thrown. OOM and ThreadDeath are 2 examples of Errors to re-throw
Review Comment:
nit. put comment on line above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18217) shutdownhookmanager should not be multithreaded (deadlock possible)
[ https://issues.apache.org/jira/browse/HADOOP-18217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530644#comment-17530644 ] Catherinot Remi commented on HADOOP-18217: -- Hi, I've added a pull request for the ExitUtil part. I've kept this JIRA's ID for that. Regards > shutdownhookmanager should not be multithreaded (deadlock possible) > --- > > Key: HADOOP-18217 > URL: https://issues.apache.org/jira/browse/HADOOP-18217 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.10.1 > Environment: linux, windows, any version >Reporter: Catherinot Remi >Priority: Minor > Labels: pull-request-available > Attachments: wtf.java > > Time Spent: 10m > Remaining Estimate: 0h > > the ShutdownHookManager class uses an executor to run hooks to have a > "timeout" notion around them. It does this using a single threaded executor. > It can leads to deadlock leaving a never-shutting-down JVM with this > execution flow: > * JVM need to exit (only daemon threads remaining or someone called > System.exit) > * ShutdowHookManager kicks in > * SHMngr executor start running some hooks > * SHMngr executor thread kicks in and, as a side effect, run some code from > one of the hook that calls System.exit (as a side effect from an external lib > for example) > * the executor thread is waiting for a lock because another thread already > entered System.exit and has its internal lock, so the executor never returns. > * SHMngr never returns > * 1st call to System.exit never returns > * JVM stuck > > using an executor with a single thread does "fake" timeouts (the task keeps > running, you can interrupt it but until it stumble upon some piece of code > that is interruptible (like an IO) it will keep running) especially since the > executor is a single threaded one. So it has this bug for example : > * caller submit 1st hook (bad one that would need 1 hour of runtime and that > cannot be interrupted) > * executor start 1st hook > * caller of the future 1st hook result timeout > * caller submit 2nd hook > * bug : 1 hook still running, 2nd hook triggers a timeout but never got the > chance to run anyway, so 1st faulty hook makes it impossible for any other > hook to have a chance to run, so running hooks in a single separate thread > does not allow to run other hooks in parallel to long ones. > > If we really really want to timeout the JVM shutdown, even accepting maybe > dirty shutdown, it should rather handle the hooks inside the initial thread > (not spawning new one(s) so not triggering the deadlock described on the 1st > place) and if a timeout was configured, only spawn a single parallel daemon > thread that sleeps the timeout delay, and then use Runtime.halt (which bypass > the hook system so should not trigger the deadlock). If the normal > System.exit ends before the timeout delay everything is fine. If the > System.exit took to much time, the JVM is killed and so the reason why this > multithreaded shutdown hook implementation was created is satisfied (avoding > having hanging JVMs) > > Had the bug with both oracle and open jdk builds, all in 1.8 major version. > hadoop 2.6 and 2.7 did not have the issue because they do not run hooks in > another thread > > Another solution is of course to configure the timeout AND to have as many > threads as needed to run the hooks so to have at least some gain to offset > the pain of the dealock scenario > > EDIT: added some logs and reproduced the problem. in fact it is located after > triggering all the hook entries and before shutting down the executor. > Current code, after running the hooks, creates a new Configuration object and > reads the configured timeout from it, applies this timeout to shutdown the > executor. I sometimes run with a classloader doing remote classloading, > Configuration loads its content using this classloader, so when shutting down > the JVM and some network error occurs the classloader fails to load the > ressources needed by Configuration. So the code crash before shutting down > the executor and ends up inside the thread's default uncaught throwable > handler, which was calling System.exit, so got stuck, so shutting down the > executor never returned, so does the JVM. > So, forget about the halt stuff (even if it is a last ressort very robust > safety net). Still I'll do a small adjustement to the final executor shutdown > code to be slightly more robust to even the strangest exceptions/errors it > encounters. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail:
[jira] [Work logged] (HADOOP-18217) shutdownhookmanager should not be multithreaded (deadlock possible)
[ https://issues.apache.org/jira/browse/HADOOP-18217?focusedWorklogId=764877=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-764877 ] ASF GitHub Bot logged work on HADOOP-18217: --- Author: ASF GitHub Bot Created on: 02/May/22 08:39 Start Date: 02/May/22 08:39 Worklog Time Spent: 10m Work Description: HerCath opened a new pull request, #4255: URL: https://github.com/apache/hadoop/pull/4255 …king halt + enlarged catches (robustness) to all Throwables (not just Exceptions) ### Description of PR I've reduced the synchronized blocks scope so System.exit and Runtime.halt calls aren't within their boundaries, so ExitUtil wrappers do not block each others (System.exit never returns if called and no SecurityException is raised, so ExitUtil.terminate was never releasing the acquired lock, thus forbidding ExitUtil.halt to be able to halt the JVM even when in the middle of a graceful shutdown, thus it was not behaving like the 2 wrapped java's methods System.exit/Runtime.halt which do not block each other) I've altered throwable handling: - what is catched: was nothing or only Exception, now all Throwables are catched (even ThreadDeath) - what is rethrown: when exit/halt has been disabled, if what was catched is an Error it will be rethrown rather than the initial ExitException/HaltException. Other Throwables will be added as suppressed to the Exit/HaltException - what wasn't catched: if not disabled, even is something was raised that wasn't catched before, it is now catched and System.exit/Runtime.halt is always called - what is suppressed: if the what needs to be rethrown is changed on the way, the newly to-be-thrown will have the old one as a suppressed Throwable. I've also done this for the Exit/Halt Exception that can supress Throwables that are not Error (might not be a so good idea) ### How was this patch tested? No more tests than the existing ones (if any). This case is not really hard to reproduce but the test would need to exit a JVM. I've not added such tests because if unit does not fork, it would kills the test suite (thus impacting all tests). I think developing a robust test for this specific case is way more hard and dangerous to offset the cost of a review, the risk of what could be missed by this review. Easiest way can be reproduced the initial bug: having a shutdown hook call ExitUtil.terminate, have another thread that will call ExitUtil.halt after (use pauses to ensure it calls it after the hook), witness the JVM not stopping and needing either an external kill or a internal Runtime.halt call, maybe check the JVM threads' stacks too to view the ExitUtil.terminate call stuck on System.exit, and ExitUtil.halt call stuck on ExitUtil.terminate. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? Issue Time Tracking --- Worklog Id: (was: 764877) Remaining Estimate: 0h Time Spent: 10m > shutdownhookmanager should not be multithreaded (deadlock possible) > --- > > Key: HADOOP-18217 > URL: https://issues.apache.org/jira/browse/HADOOP-18217 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.10.1 > Environment: linux, windows, any version >Reporter: Catherinot Remi >Priority: Minor > Attachments: wtf.java > > Time Spent: 10m > Remaining Estimate: 0h > > the ShutdownHookManager class uses an executor to run hooks to have a > "timeout" notion around them. It does this using a single threaded executor. > It can leads to deadlock leaving a never-shutting-down JVM with this > execution flow: > * JVM need to exit (only daemon threads remaining or someone called > System.exit) > * ShutdowHookManager kicks in > * SHMngr executor start running some hooks > * SHMngr executor thread kicks in and, as a side effect, run some code from > one of the hook that calls System.exit (as a side effect from an external lib > for example) > * the executor thread is waiting for a lock because another thread already > entered System.exit and has its internal lock, so the executor never returns. > * SHMngr
[jira] [Updated] (HADOOP-18217) shutdownhookmanager should not be multithreaded (deadlock possible)
[ https://issues.apache.org/jira/browse/HADOOP-18217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18217: Labels: pull-request-available (was: ) > shutdownhookmanager should not be multithreaded (deadlock possible) > --- > > Key: HADOOP-18217 > URL: https://issues.apache.org/jira/browse/HADOOP-18217 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.10.1 > Environment: linux, windows, any version >Reporter: Catherinot Remi >Priority: Minor > Labels: pull-request-available > Attachments: wtf.java > > Time Spent: 10m > Remaining Estimate: 0h > > the ShutdownHookManager class uses an executor to run hooks to have a > "timeout" notion around them. It does this using a single threaded executor. > It can leads to deadlock leaving a never-shutting-down JVM with this > execution flow: > * JVM need to exit (only daemon threads remaining or someone called > System.exit) > * ShutdowHookManager kicks in > * SHMngr executor start running some hooks > * SHMngr executor thread kicks in and, as a side effect, run some code from > one of the hook that calls System.exit (as a side effect from an external lib > for example) > * the executor thread is waiting for a lock because another thread already > entered System.exit and has its internal lock, so the executor never returns. > * SHMngr never returns > * 1st call to System.exit never returns > * JVM stuck > > using an executor with a single thread does "fake" timeouts (the task keeps > running, you can interrupt it but until it stumble upon some piece of code > that is interruptible (like an IO) it will keep running) especially since the > executor is a single threaded one. So it has this bug for example : > * caller submit 1st hook (bad one that would need 1 hour of runtime and that > cannot be interrupted) > * executor start 1st hook > * caller of the future 1st hook result timeout > * caller submit 2nd hook > * bug : 1 hook still running, 2nd hook triggers a timeout but never got the > chance to run anyway, so 1st faulty hook makes it impossible for any other > hook to have a chance to run, so running hooks in a single separate thread > does not allow to run other hooks in parallel to long ones. > > If we really really want to timeout the JVM shutdown, even accepting maybe > dirty shutdown, it should rather handle the hooks inside the initial thread > (not spawning new one(s) so not triggering the deadlock described on the 1st > place) and if a timeout was configured, only spawn a single parallel daemon > thread that sleeps the timeout delay, and then use Runtime.halt (which bypass > the hook system so should not trigger the deadlock). If the normal > System.exit ends before the timeout delay everything is fine. If the > System.exit took to much time, the JVM is killed and so the reason why this > multithreaded shutdown hook implementation was created is satisfied (avoding > having hanging JVMs) > > Had the bug with both oracle and open jdk builds, all in 1.8 major version. > hadoop 2.6 and 2.7 did not have the issue because they do not run hooks in > another thread > > Another solution is of course to configure the timeout AND to have as many > threads as needed to run the hooks so to have at least some gain to offset > the pain of the dealock scenario > > EDIT: added some logs and reproduced the problem. in fact it is located after > triggering all the hook entries and before shutting down the executor. > Current code, after running the hooks, creates a new Configuration object and > reads the configured timeout from it, applies this timeout to shutdown the > executor. I sometimes run with a classloader doing remote classloading, > Configuration loads its content using this classloader, so when shutting down > the JVM and some network error occurs the classloader fails to load the > ressources needed by Configuration. So the code crash before shutting down > the executor and ends up inside the thread's default uncaught throwable > handler, which was calling System.exit, so got stuck, so shutting down the > executor never returned, so does the JVM. > So, forget about the halt stuff (even if it is a last ressort very robust > safety net). Still I'll do a small adjustement to the final executor shutdown > code to be slightly more robust to even the strangest exceptions/errors it > encounters. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] HerCath opened a new pull request, #4255: HADOOP-18217. ExitUtil synchronized blocks reduced to avoid exit bloc…
HerCath opened a new pull request, #4255: URL: https://github.com/apache/hadoop/pull/4255 …king halt + enlarged catches (robustness) to all Throwables (not just Exceptions) ### Description of PR I've reduced the synchronized blocks scope so System.exit and Runtime.halt calls aren't within their boundaries, so ExitUtil wrappers do not block each others (System.exit never returns if called and no SecurityException is raised, so ExitUtil.terminate was never releasing the acquired lock, thus forbidding ExitUtil.halt to be able to halt the JVM even when in the middle of a graceful shutdown, thus it was not behaving like the 2 wrapped java's methods System.exit/Runtime.halt which do not block each other) I've altered throwable handling: - what is catched: was nothing or only Exception, now all Throwables are catched (even ThreadDeath) - what is rethrown: when exit/halt has been disabled, if what was catched is an Error it will be rethrown rather than the initial ExitException/HaltException. Other Throwables will be added as suppressed to the Exit/HaltException - what wasn't catched: if not disabled, even is something was raised that wasn't catched before, it is now catched and System.exit/Runtime.halt is always called - what is suppressed: if the what needs to be rethrown is changed on the way, the newly to-be-thrown will have the old one as a suppressed Throwable. I've also done this for the Exit/Halt Exception that can supress Throwables that are not Error (might not be a so good idea) ### How was this patch tested? No more tests than the existing ones (if any). This case is not really hard to reproduce but the test would need to exit a JVM. I've not added such tests because if unit does not fork, it would kills the test suite (thus impacting all tests). I think developing a robust test for this specific case is way more hard and dangerous to offset the cost of a review, the risk of what could be missed by this review. Easiest way can be reproduced the initial bug: having a shutdown hook call ExitUtil.terminate, have another thread that will call ExitUtil.halt after (use pauses to ensure it calls it after the hook), witness the JVM not stopping and needing either an external kill or a internal Runtime.halt call, maybe check the JVM threads' stacks too to view the ExitUtil.terminate call stuck on System.exit, and ExitUtil.halt call stuck on ExitUtil.terminate. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4250: YARN-11121:Conditional judgment Add in getClusterMetrics
hadoop-yetus commented on PR #4250:
URL: https://github.com/apache/hadoop/pull/4250#issuecomment-1114604502
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 2s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 41s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4250/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router:
The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 0m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 2m 3s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4250/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt)
| hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 102m 27s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.router.webapp.TestRouterWebServicesREST |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4250/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4250 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux e4a00d5eda7f 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 31492ab1deb158708ddecba627c3af0f6bae5f75 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
[jira] [Work logged] (HADOOP-18219) Fix shadedclient test failure
[
https://issues.apache.org/jira/browse/HADOOP-18219?focusedWorklogId=764873=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-764873
]
ASF GitHub Bot logged work on HADOOP-18219:
---
Author: ASF GitHub Bot
Created on: 02/May/22 08:02
Start Date: 02/May/22 08:02
Worklog Time Spent: 10m
Work Description: GauthamBanasandra commented on PR #4254:
URL: https://github.com/apache/hadoop/pull/4254#issuecomment-1114595025
Thanks @aajisaka. I was able to verify that this PR fixes the issue
[build.log](https://github.com/apache/hadoop/files/8601569/build.log)
.
Issue Time Tracking
---
Worklog Id: (was: 764873)
Time Spent: 1h 10m (was: 1h)
> Fix shadedclient test failure
> -
>
> Key: HADOOP-18219
> URL: https://issues.apache.org/jira/browse/HADOOP-18219
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.4.0
> Environment: Debian 10
>Reporter: Gautham Banasandra
>Assignee: PJ Fanning
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Two of the shaded client tests are failing on Debian 10 ever since this
> commit -
> https://github.com/apache/hadoop/commit/63187083cc3b9bb1c1e90e692e271958561f9cc8.
> The failures are as follows -
> 1st test failure -
> {code}
> [INFO] Running org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
> 18.315 s <<< FAILURE! - in org.apache.hadoop.example.ITUseMiniCluster
> [ERROR] useWebHDFS(org.apache.hadoop.example.ITUseMiniCluster) Time elapsed:
> 12.048 s <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.startResourceManager(MiniYARNCluster.java:384)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.access$300(MiniYARNCluster.java:129)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:500)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:123)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceStart(MiniYARNCluster.java:333)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
> at
> org.apache.hadoop.example.ITUseMiniCluster.clusterUp(ITUseMiniCluster.java:84)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
>
[GitHub] [hadoop] GauthamBanasandra commented on pull request #4254: HADOOP-18219. Fix shadedclient test failure
GauthamBanasandra commented on PR #4254: URL: https://github.com/apache/hadoop/pull/4254#issuecomment-1114595025 Thanks @aajisaka. I was able to verify that this PR fixes the issue [build.log](https://github.com/apache/hadoop/files/8601569/build.log) . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4250: YARN-11121:Conditional judgment Add in getClusterMetrics
slfan1989 commented on PR #4250:
URL: https://github.com/apache/hadoop/pull/4250#issuecomment-1114552633
I need to ask a question, is it necessary to use a multi-threaded method
when acquiring sub-clusters, can it not be traversed directly through a for
loop? I tried to rewrite part of the code, can you help me to review it.
`Map subclusters =
federationFacade.getSubClusters(true);
Set subClusterIds = subclusters.keySet();
Map clusterMetrics =
Maps.newHashMap();
for (SubClusterId subClusterId : subClusterIds) {
ApplicationClientProtocol protocol =
getClientRMProxyForSubCluster(subClusterId);
GetClusterMetricsResponse metricsResponse =
protocol.getClusterMetrics(request);
clusterMetrics.put(subClusterId,metricsResponse);
}
return RouterYarnClientUtils.merge(clusterMetrics.values());`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ashutoshcipher commented on pull request #4244: YARN-11119. Backport YARN-10538 to branch-2.10
ashutoshcipher commented on PR #4244: URL: https://github.com/apache/hadoop/pull/4244#issuecomment-1114538988 @aajisaka - Please review this PR. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
