[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[ https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17801116#comment-17801116 ] ASF GitHub Bot commented on HDFS-17223: --- gp1314 commented on PR #6183: URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1871841096 Unfortunately, I didn't reproduce the problem. In the past, stopping a JN and restarting NN took a long time to initialize. I will pay more attention to the root cause of the problem. > Add journalnode maintenance node list > - > > Key: HDFS-17223 > URL: https://issues.apache.org/jira/browse/HDFS-17223 > Project: Hadoop HDFS > Issue Type: Improvement > Components: qjm >Affects Versions: 3.3.6 >Reporter: kuper >Priority: Major > Labels: pull-request-available > > * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes > are available and 1 journal node fails to start due to machine issues, it > will result in a long initialization time for the namenode (around 30-40 > minutes, depending on the IPC timeout and retry policy configuration). > * The failed journal node cannot recover immediately, but HDFS can still > function in this situation. In our production environment, we encountered > this issue and had to reduce the IPC timeout and adjust the retry policy to > accelerate the namenode initialization and provide services. > * I'm wondering if it would be possible to have a journal node maintenance > list to speed up the namenode initialization knowing that one journal node > cannot provide services in advance? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[ https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17794523#comment-17794523 ] ASF GitHub Bot commented on HDFS-17223: --- xinglin commented on PR #6183: URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1846575811 My understanding is a similar issue is happening here as what I tried to fix in [HDFS-17030](https://issues.apache.org/jira/browse/HDFS-17030): when a JN is not responsive (either it is down or it hangs), the starting NN would try to connect to it anyway with retries. Thus, it would wait for `ipc.client.connect.timeout` * `ipc.client.connect.max.retries.on.timeouts` when NN is not able to establish a socket to the journal node, or `ipc.client.rpc-timeout.ms` when a socket is established but the journal node fails to send back a response. > Add journalnode maintenance node list > - > > Key: HDFS-17223 > URL: https://issues.apache.org/jira/browse/HDFS-17223 > Project: Hadoop HDFS > Issue Type: Improvement > Components: qjm >Affects Versions: 3.3.6 >Reporter: kuper >Priority: Major > Labels: pull-request-available > > * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes > are available and 1 journal node fails to start due to machine issues, it > will result in a long initialization time for the namenode (around 30-40 > minutes, depending on the IPC timeout and retry policy configuration). > * The failed journal node cannot recover immediately, but HDFS can still > function in this situation. In our production environment, we encountered > this issue and had to reduce the IPC timeout and adjust the retry policy to > accelerate the namenode initialization and provide services. > * I'm wondering if it would be possible to have a journal node maintenance > list to speed up the namenode initialization knowing that one journal node > cannot provide services in advance? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[ https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17790079#comment-17790079 ] ASF GitHub Bot commented on HDFS-17223: --- Hexiaoqiao commented on PR #6183: URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1827820008 Thanks @gp1314 and @xinglin for your works. I am not very sure to get the total purpose here. > In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes are available and 1 journal node fails to start due to machine issues, it will result in a long initialization time for the namenode (around 30-40 minutes, depending on the IPC timeout and retry policy configuration). Do you mean that NameNode restart will cost extra over 30~40 minutes while 1/3 JN could not be available? It is interesting where it costs? IIUC, The majority JN work well, it will connect and interact well. > The failed journal node cannot recover immediately, but HDFS can still function in this situation. In our production environment, we encountered this issue and had to reduce the IPC timeout and adjust the retry policy to accelerate the namenode initialization and provide services. I used to maintain JNs online one by one, but didn't meet timeout at NameNode side. Not sure what different between them, one point is the version could have some differences(our version is based on 2.7.1 with some internal improvement.) Thanks again. Please correct me if I missed something. > Add journalnode maintenance node list > - > > Key: HDFS-17223 > URL: https://issues.apache.org/jira/browse/HDFS-17223 > Project: Hadoop HDFS > Issue Type: Improvement > Components: qjm >Affects Versions: 3.3.6 >Reporter: kuper >Priority: Major > Labels: pull-request-available > > * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes > are available and 1 journal node fails to start due to machine issues, it > will result in a long initialization time for the namenode (around 30-40 > minutes, depending on the IPC timeout and retry policy configuration). > * The failed journal node cannot recover immediately, but HDFS can still > function in this situation. In our production environment, we encountered > this issue and had to reduce the IPC timeout and adjust the retry policy to > accelerate the namenode initialization and provide services. > * I'm wondering if it would be possible to have a journal node maintenance > list to speed up the namenode initialization knowing that one journal node > cannot provide services in advance? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[ https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17789041#comment-17789041 ] ASF GitHub Bot commented on HDFS-17223: --- gp1314 commented on PR #6183: URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1824082862 @Hexiaoqiao , could you help to review this improvement in the code? If it's feasible, can it be merged into the trunk branch? I would be very grateful. > Add journalnode maintenance node list > - > > Key: HDFS-17223 > URL: https://issues.apache.org/jira/browse/HDFS-17223 > Project: Hadoop HDFS > Issue Type: Improvement > Components: qjm >Affects Versions: 3.3.6 >Reporter: kuper >Priority: Major > Labels: pull-request-available > > * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes > are available and 1 journal node fails to start due to machine issues, it > will result in a long initialization time for the namenode (around 30-40 > minutes, depending on the IPC timeout and retry policy configuration). > * The failed journal node cannot recover immediately, but HDFS can still > function in this situation. In our production environment, we encountered > this issue and had to reduce the IPC timeout and adjust the retry policy to > accelerate the namenode initialization and provide services. > * I'm wondering if it would be possible to have a journal node maintenance > list to speed up the namenode initialization knowing that one journal node > cannot provide services in advance? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[ https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788603#comment-17788603 ] ASF GitHub Bot commented on HDFS-17223: --- gp1314 commented on PR #6183: URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1821996456 @xinglin , thank you very much for code review and suggestions, I will pay attention next time. > Add journalnode maintenance node list > - > > Key: HDFS-17223 > URL: https://issues.apache.org/jira/browse/HDFS-17223 > Project: Hadoop HDFS > Issue Type: Improvement > Components: qjm >Affects Versions: 3.3.6 >Reporter: kuper >Priority: Major > Labels: pull-request-available > > * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes > are available and 1 journal node fails to start due to machine issues, it > will result in a long initialization time for the namenode (around 30-40 > minutes, depending on the IPC timeout and retry policy configuration). > * The failed journal node cannot recover immediately, but HDFS can still > function in this situation. In our production environment, we encountered > this issue and had to reduce the IPC timeout and adjust the retry policy to > accelerate the namenode initialization and provide services. > * I'm wondering if it would be possible to have a journal node maintenance > list to speed up the namenode initialization knowing that one journal node > cannot provide services in advance? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[ https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788587#comment-17788587 ] ASF GitHub Bot commented on HDFS-17223: --- xinglin commented on PR #6183: URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1821934677 @gp1314, can you try not force-push next time? Just push incremental commits so that I can review the new change. thanks, > Add journalnode maintenance node list > - > > Key: HDFS-17223 > URL: https://issues.apache.org/jira/browse/HDFS-17223 > Project: Hadoop HDFS > Issue Type: Improvement > Components: qjm >Affects Versions: 3.3.6 >Reporter: kuper >Priority: Major > Labels: pull-request-available > > * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes > are available and 1 journal node fails to start due to machine issues, it > will result in a long initialization time for the namenode (around 30-40 > minutes, depending on the IPC timeout and retry policy configuration). > * The failed journal node cannot recover immediately, but HDFS can still > function in this situation. In our production environment, we encountered > this issue and had to reduce the IPC timeout and adjust the retry policy to > accelerate the namenode initialization and provide services. > * I'm wondering if it would be possible to have a journal node maintenance > list to speed up the namenode initialization knowing that one journal node > cannot provide services in advance? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788488#comment-17788488
]
ASF GitHub Bot commented on HDFS-17223:
---
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1821360101
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 59s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 38s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 2s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/12/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 322 unchanged
- 1 fixed = 330 total (was 323) |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 46s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 249m 12s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/12/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 401m 6s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6183 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 35408deec17d 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8c5e2b37510c2adb4aef1afa8d2c94daed0d0a0c |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788351#comment-17788351
]
ASF GitHub Bot commented on HDFS-17223:
---
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1820590066
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 34s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 3s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/11/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 322 unchanged
- 1 fixed = 330 total (was 323) |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 250m 18s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 402m 38s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6183 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux fb231206f795 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1046925340b69069b09ea82de5a64bf01b46d08c |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/11/testReport/ |
| Max. process+thread count | 2688 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788254#comment-17788254
]
ASF GitHub Bot commented on HDFS-17223:
---
xinglin commented on code in PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#discussion_r1400039039
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java:
##
@@ -1171,6 +1179,47 @@ public void testSelectViaRpcAfterJNRestart() throws
Exception {
}
}
+ /**
+ * Tests to throw an exception if the jn maintenance nodes exceeds half of
the journalnode number.
Review Comment:
nit:
"if the jn maintenance nodes exceeds half of the journalnode number." ->
"when more than half of the journal nodes are in maintenance mode".
> Add journalnode maintenance node list
> -
>
> Key: HDFS-17223
> URL: https://issues.apache.org/jira/browse/HDFS-17223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: qjm
>Affects Versions: 3.3.6
>Reporter: kuper
>Priority: Major
> Labels: pull-request-available
>
> * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes
> are available and 1 journal node fails to start due to machine issues, it
> will result in a long initialization time for the namenode (around 30-40
> minutes, depending on the IPC timeout and retry policy configuration).
> * The failed journal node cannot recover immediately, but HDFS can still
> function in this situation. In our production environment, we encountered
> this issue and had to reduce the IPC timeout and adjust the retry policy to
> accelerate the namenode initialization and provide services.
> * I'm wondering if it would be possible to have a journal node maintenance
> list to speed up the namenode initialization knowing that one journal node
> cannot provide services in advance?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787913#comment-17787913
]
ASF GitHub Bot commented on HDFS-17223:
---
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1818691838
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 16m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 33s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/10/artifact/out/blanks-eol.txt)
| The patch has 6 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 4s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/10/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 322 unchanged
- 1 fixed = 330 total (was 323) |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 248m 28s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 417m 55s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6183 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 5b0c9ebc7458 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / dc94c93398be3a6559898faa377dc7b8f9091244 |
| Default Java |
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17786000#comment-17786000
]
ASF GitHub Bot commented on HDFS-17223:
---
xinglin commented on code in PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#discussion_r1392945651
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:
##
@@ -667,6 +700,9 @@ AsyncLoggerSet getLoggerSetForTests() {
@Override
public void doPreUpgrade() throws IOException {
+if (isEnableJnMaintenance()) {
+ throw new IOException("doPreUpgrade() does not support enabling jn
maintenance mode");
Review Comment:
nit: doPreUpgrade() does not support enabling jn maintenance mode ->
doPreUpgrade() is not support while in jn maintenance mode
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:
##
@@ -684,6 +720,9 @@ public void doPreUpgrade() throws IOException {
@Override
public void doUpgrade(Storage storage) throws IOException {
+if (isEnableJnMaintenance()) {
+ throw new IOException("doUpgrade() does not support enabling jn
maintenance mode");
Review Comment:
same here.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml:
##
@@ -6333,6 +6333,20 @@
+
+dfs.journalnode.maintenance.nodes
+
+
+ In the case of one out of three journal nodes being down, theoretically
the service can still
+ continue. However, in reality, the downed node may not recover quickly.
If the Namenode needs
+ to be restarted, it will try the downed journal node through the lengthy
RPC retry mechanism,
+ resulting in a long initialization time for the Namenode to provide
services. By adding the
+ downed journal node to the maintenance nodes, the initialization time of
the Namenode in such
+ scenarios can be accelerated.
Review Comment:
nit
->
In the case that one out of three journal nodes being down, theoretically
HDFS can still
function. However, in reality, the unavailable journal node may not
recover quickly. During this period, when we need to restart an Namenode, the
Namenode will try to connect to the unavailable journal node through the
lengthy RPC retry mechanism,
resulting in a long initialization time for the Namenode. By adding
these
unavailable journal nodes to the maintenance nodes, we will skip these
unavailable journal nodes during Namenode initialization and thus reduce
namenode startup time.
1-node example values: <>
2-node example values: <>
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java:
##
@@ -1137,4 +1139,24 @@ public void testAddTransferRateMetricForInvalidValue() {
DFSUtil.addTransferRateMetric(mockMetrics, 100, 0);
verify(mockMetrics, times(0)).addReadTransferRate(anyLong());
}
+
+ @Test
+ public void testGetHostSet() {
+String[] testAddrs = new String[] {NS1_NN_ADDR, NS1_NN1_ADDR};
Review Comment:
this test case is a bit confusing. Can we just use
"unreachable-host1.com:9000" instead?
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQuorumJournalManager.java:
##
@@ -1171,6 +1176,21 @@ public void testSelectViaRpcAfterJNRestart() throws
Exception {
}
}
+ /**
+ * Tests to throw an exception if the jn maintenance nodes exceeds half of
the journalnode number.
+ */
+ @Test
+ public void testJNMaintenanceListViaRpcTwoJNsError() throws Exception {
+StringJoiner maintenanceListBuff = new StringJoiner(",");
+for (int i = 0; i < 2; i++) {
+ maintenanceListBuff.add(
+
NetUtils.getHostPortString(cluster.getJournalNode(i).getBoundIpcAddress()));
+}
+this.conf.set(DFS_JOURNALNODE_MAINTENANCE_NODES_KEY,
maintenanceListBuff.toString());
+assertThrows(IllegalArgumentException.class, () -> createSpyingQJM());
+ }
+
+
Review Comment:
I assume we also need a test case to to verify maintenance journal nodes are
indeed excluded? total JN=3, with 1 is excluded. after we initialize, verify
only 2 JNs are included.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:
##
@@ -701,6 +740,9 @@ public void doUpgrade(Storage storage) throws IOException {
@Override
public void doFinalize() throws IOException {
+if (isEnableJnMaintenance()) {
+ throw new IOException("doFinalize() does not support enabling jn
maintenance mode");
Review Comment:
same here. and for others functions.
> Add journalnode maintenance node list
> -
>
> Key: HDFS-17223
> URL
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785987#comment-17785987
]
ASF GitHub Bot commented on HDFS-17223:
---
xinglin commented on code in PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#discussion_r1392944771
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:
##
@@ -406,21 +421,39 @@ private void recoverUnclosedSegment(long segmentTxId)
throws IOException {
logToSync.getStartTxId(),
logToSync.getEndTxId()));
}
-
- static List createLoggers(Configuration conf,
+
+ List createLoggers(Configuration conf,
+ URI uri,
+ NamespaceInfo nsInfo,
+ AsyncLogger.Factory factory,
+ String nameServiceId)
+ throws IOException {
+String[] skipNodesHostPort = conf.getTrimmedStrings(
+DFS_JOURNALNODE_MAINTENANCE_NODES_KEY,
DFS_JOURNALNODE_MAINTENANCE_NODES_DEFAULT);
+return createLoggers(conf, uri, nsInfo, factory, nameServiceId,
skipNodesHostPort);
+ }
+
+ private List createLoggers(Configuration conf,
URI uri,
NamespaceInfo nsInfo,
AsyncLogger.Factory factory,
- String nameServiceId)
+ String nameServiceId,
+ String[] skipNodesHostPort)
throws IOException {
List ret = Lists.newArrayList();
List addrs = Util.getAddressesList(uri, conf);
if (addrs.size() % 2 == 0) {
LOG.warn("Quorum journal URI '" + uri + "' has an even number " +
"of Journal Nodes specified. This is not recommended!");
}
+setQuorumJournalCount(addrs.size());
+HostSet skipSet = DFSUtil.getHostSet(skipNodesHostPort);
String jid = parseJournalId(uri);
for (InetSocketAddress addr : addrs) {
+ if(skipSet.match(addr)) {
+LOG.info("The node {} is a maintenance node and will skip
initialization.", addr);
Review Comment:
nit: "will skip initialization" -> "will be skipped"
> Add journalnode maintenance node list
> -
>
> Key: HDFS-17223
> URL: https://issues.apache.org/jira/browse/HDFS-17223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: qjm
>Affects Versions: 3.3.6
>Reporter: kuper
>Priority: Major
> Labels: pull-request-available
>
> * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes
> are available and 1 journal node fails to start due to machine issues, it
> will result in a long initialization time for the namenode (around 30-40
> minutes, depending on the IPC timeout and retry policy configuration).
> * The failed journal node cannot recover immediately, but HDFS can still
> function in this situation. In our production environment, we encountered
> this issue and had to reduce the IPC timeout and adjust the retry policy to
> accelerate the namenode initialization and provide services.
> * I'm wondering if it would be possible to have a journal node maintenance
> list to speed up the namenode initialization knowing that one journal node
> cannot provide services in advance?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785986#comment-17785986
]
ASF GitHub Bot commented on HDFS-17223:
---
xinglin commented on code in PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#discussion_r1392943712
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java:
##
@@ -250,6 +262,9 @@ Map
createNewUniqueEpoch()
@Override
public void format(NamespaceInfo nsInfo, boolean force) throws IOException {
+if (isEnableJnMaintenance()) {
+ throw new IOException("format() does not support enabling jn maintenance
mode");
Review Comment:
nit:
`isEnableJnMaintenance(`) -> `isJNMainteanceEnabled()` or should it actually
be `IsJNInMaintenanceMode()` ?
"format() does not support enabling jn maintenance mode" -> "Formatting a
journal node is not supported while in JN maintenance mode"
> Add journalnode maintenance node list
> -
>
> Key: HDFS-17223
> URL: https://issues.apache.org/jira/browse/HDFS-17223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: qjm
>Affects Versions: 3.3.6
>Reporter: kuper
>Priority: Major
> Labels: pull-request-available
>
> * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes
> are available and 1 journal node fails to start due to machine issues, it
> will result in a long initialization time for the namenode (around 30-40
> minutes, depending on the IPC timeout and retry policy configuration).
> * The failed journal node cannot recover immediately, but HDFS can still
> function in this situation. In our production environment, we encountered
> this issue and had to reduce the IPC timeout and adjust the retry policy to
> accelerate the namenode initialization and provide services.
> * I'm wondering if it would be possible to have a journal node maintenance
> list to speed up the namenode initialization knowing that one journal node
> cannot provide services in advance?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[ https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785981#comment-17785981 ] ASF GitHub Bot commented on HDFS-17223: --- xinglin commented on code in PR #6183: URL: https://github.com/apache/hadoop/pull/6183#discussion_r1392934193 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java: ## @@ -144,7 +149,14 @@ public QuorumJournalManager(Configuration conf, this.uri = uri; this.nsInfo = nsInfo; this.nameServiceId = nameServiceId; -this.loggers = new AsyncLoggerSet(createLoggers(loggerFactory)); +this.loggers = new AsyncLoggerSet(createLoggers(loggerFactory), this.quorumJournalCount); Review Comment: this.quorumJournalCount is set inside createLoggers(). Can we separate them as two separate steps, like the following? ``` // Keep this in comment: // createLoggers() will set quorumJournalCount to total number of journal nodes while return a list of healthy/good journal nodes. List loggers = createLoggers(loggerFactory); this.loggers = new AsyncLoggerSet(loggers, this.quorumJournalCount); ``` > Add journalnode maintenance node list > - > > Key: HDFS-17223 > URL: https://issues.apache.org/jira/browse/HDFS-17223 > Project: Hadoop HDFS > Issue Type: Improvement > Components: qjm >Affects Versions: 3.3.6 >Reporter: kuper >Priority: Major > Labels: pull-request-available > > * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes > are available and 1 journal node fails to start due to machine issues, it > will result in a long initialization time for the namenode (around 30-40 > minutes, depending on the IPC timeout and retry policy configuration). > * The failed journal node cannot recover immediately, but HDFS can still > function in this situation. In our production environment, we encountered > this issue and had to reduce the IPC timeout and adjust the retry policy to > accelerate the namenode initialization and provide services. > * I'm wondering if it would be possible to have a journal node maintenance > list to speed up the namenode initialization knowing that one journal node > cannot provide services in advance? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785975#comment-17785975
]
ASF GitHub Bot commented on HDFS-17223:
---
xinglin commented on code in PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#discussion_r1392918855
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java:
##
@@ -1982,4 +1983,32 @@ public static void addTransferRateMetric(final
DataNodeMetrics metrics, final lo
LOG.warn("Unexpected value for data transfer bytes={} duration={}",
read, duration);
}
}
+
+ /**
+ * Construct a HostSet from an array of "ip:port" strings.
+ * @param nodesHostPort ip port string array.
+ * @return HostSet of InetSocketAddress.
+ */
+ public static HostSet getHostSet(String[] nodesHostPort) {
+HostSet retSet = new HostSet();
+for (String hostPort : nodesHostPort) {
+ try {
+URI uri = new URI("dummy", hostPort, null, null, null);
+int port = uri.getPort();
+if (port == -1 || port == 0) {
Review Comment:
port=0 is valid. I guess we should probably just check for -1 here?
```
/**
* Returns the port number of this URI.
*
* The port component of a URI, if defined, is a non-negative
* integer.
*
* @return The port component of this URI,
* or {@code -1} if the port is undefined
*/
public int getPort() {
return port;
}
```
> Add journalnode maintenance node list
> -
>
> Key: HDFS-17223
> URL: https://issues.apache.org/jira/browse/HDFS-17223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: qjm
>Affects Versions: 3.3.6
>Reporter: kuper
>Priority: Major
> Labels: pull-request-available
>
> * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes
> are available and 1 journal node fails to start due to machine issues, it
> will result in a long initialization time for the namenode (around 30-40
> minutes, depending on the IPC timeout and retry policy configuration).
> * The failed journal node cannot recover immediately, but HDFS can still
> function in this situation. In our production environment, we encountered
> this issue and had to reduce the IPC timeout and adjust the retry policy to
> accelerate the namenode initialization and provide services.
> * I'm wondering if it would be possible to have a journal node maintenance
> list to speed up the namenode initialization knowing that one journal node
> cannot provide services in advance?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785919#comment-17785919
]
ASF GitHub Bot commented on HDFS-17223:
---
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1810342572
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 5m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 50s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 31s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/9/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 322 unchanged
- 1 fixed = 330 total (was 323) |
| +1 :green_heart: | mvnsite | 0m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 207m 31s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/9/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | | The patch does not
generate ASF License warnings. |
| | | 305m 26s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithDecoding |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestDecommissionWithStriped |
| | hadoop.hdfs.TestCrcCorruption |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6183 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 8d6598f059e3 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revisio
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785917#comment-17785917
]
ASF GitHub Bot commented on HDFS-17223:
---
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1810339618
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 29s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 6s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 35s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/8/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 322 unchanged
- 1 fixed = 330 total (was 323) |
| +1 :green_heart: | mvnsite | 0m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 3s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 206m 12s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 305m 54s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestDecommissionWithStriped |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.TestDFSInputStreamBlockLocations |
| | hadoop.hdfs.TestDFSStripedOutputStream |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6183 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux be68a6bc0f70 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 711aa42d79832c191747b2be15e2985577a1d358 |
| Default
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785822#comment-17785822
]
ASF GitHub Bot commented on HDFS-17223:
---
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1809879806
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 41s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 1s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 3s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/7/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 322 unchanged
- 1 fixed = 330 total (was 323) |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 239m 36s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 396m 26s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6183 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux cabf21baac90 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5f415d73c212b0313a558ed06457c7e09a1b0e09 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-pos
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[ https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785757#comment-17785757 ] ASF GitHub Bot commented on HDFS-17223: --- xinglin commented on code in PR #6183: URL: https://github.com/apache/hadoop/pull/6183#discussion_r1392039679 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java: ## @@ -62,6 +66,7 @@ import org.apache.hadoop.classification.VisibleForTesting; import org.apache.hadoop.thirdparty.com.google.common.base.Joiner; import org.apache.hadoop.util.Preconditions; + Review Comment: remove this. > Add journalnode maintenance node list > - > > Key: HDFS-17223 > URL: https://issues.apache.org/jira/browse/HDFS-17223 > Project: Hadoop HDFS > Issue Type: Improvement > Components: qjm >Affects Versions: 3.3.6 >Reporter: kuper >Priority: Major > Labels: pull-request-available > > * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes > are available and 1 journal node fails to start due to machine issues, it > will result in a long initialization time for the namenode (around 30-40 > minutes, depending on the IPC timeout and retry policy configuration). > * The failed journal node cannot recover immediately, but HDFS can still > function in this situation. In our production environment, we encountered > this issue and had to reduce the IPC timeout and adjust the retry policy to > accelerate the namenode initialization and provide services. > * I'm wondering if it would be possible to have a journal node maintenance > list to speed up the namenode initialization knowing that one journal node > cannot provide services in advance? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785756#comment-17785756
]
ASF GitHub Bot commented on HDFS-17223:
---
xinglin commented on code in PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#discussion_r1392037472
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java:
##
@@ -1982,4 +1983,28 @@ public static void addTransferRateMetric(final
DataNodeMetrics metrics, final lo
LOG.warn("Unexpected value for data transfer bytes={} duration={}",
read, duration);
}
}
+
+ /**
+ * Retrieve InetSocketAddress set by ip port string array.
+ * @param nodesHostPort ip port string array.
+ * @return HostSet of InetSocketAddress.
+ */
+ public static HostSet convertHostSet(String[] nodesHostPort) {
+HostSet retSet = new HostSet();
+for (String hostPort : nodesHostPort) {
+ try {
+URI uri = new URI("dummy", hostPort, null, null, null);
+int port = uri.getPort() == -1 ? 0 : uri.getPort();
Review Comment:
is it appropriate here? It seems 0 is a valid port but -1 should indicate
port is not set?
> A valid port value is between 0 and 65535. A port number of zero will let
the system pick up an ephemeral port in a bind operation.
> Add journalnode maintenance node list
> -
>
> Key: HDFS-17223
> URL: https://issues.apache.org/jira/browse/HDFS-17223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: qjm
>Affects Versions: 3.3.6
>Reporter: kuper
>Priority: Major
> Labels: pull-request-available
>
> * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes
> are available and 1 journal node fails to start due to machine issues, it
> will result in a long initialization time for the namenode (around 30-40
> minutes, depending on the IPC timeout and retry policy configuration).
> * The failed journal node cannot recover immediately, but HDFS can still
> function in this situation. In our production environment, we encountered
> this issue and had to reduce the IPC timeout and adjust the retry policy to
> accelerate the namenode initialization and provide services.
> * I'm wondering if it would be possible to have a journal node maintenance
> list to speed up the namenode initialization knowing that one journal node
> cannot provide services in advance?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785755#comment-17785755
]
ASF GitHub Bot commented on HDFS-17223:
---
xinglin commented on code in PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#discussion_r1392034064
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java:
##
@@ -1982,4 +1983,28 @@ public static void addTransferRateMetric(final
DataNodeMetrics metrics, final lo
LOG.warn("Unexpected value for data transfer bytes={} duration={}",
read, duration);
}
}
+
+ /**
+ * Retrieve InetSocketAddress set by ip port string array.
+ * @param nodesHostPort ip port string array.
+ * @return HostSet of InetSocketAddress.
+ */
+ public static HostSet convertHostSet(String[] nodesHostPort) {
Review Comment:
nit: convertHostSet -> getHostSet() or ConvertToHostSet()
> Add journalnode maintenance node list
> -
>
> Key: HDFS-17223
> URL: https://issues.apache.org/jira/browse/HDFS-17223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: qjm
>Affects Versions: 3.3.6
>Reporter: kuper
>Priority: Major
> Labels: pull-request-available
>
> * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes
> are available and 1 journal node fails to start due to machine issues, it
> will result in a long initialization time for the namenode (around 30-40
> minutes, depending on the IPC timeout and retry policy configuration).
> * The failed journal node cannot recover immediately, but HDFS can still
> function in this situation. In our production environment, we encountered
> this issue and had to reduce the IPC timeout and adjust the retry policy to
> accelerate the namenode initialization and provide services.
> * I'm wondering if it would be possible to have a journal node maintenance
> list to speed up the namenode initialization knowing that one journal node
> cannot provide services in advance?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785754#comment-17785754
]
ASF GitHub Bot commented on HDFS-17223:
---
xinglin commented on code in PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#discussion_r1392032362
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java:
##
@@ -1982,4 +1983,28 @@ public static void addTransferRateMetric(final
DataNodeMetrics metrics, final lo
LOG.warn("Unexpected value for data transfer bytes={} duration={}",
read, duration);
}
}
+
+ /**
+ * Retrieve InetSocketAddress set by ip port string array.
Review Comment:
nit: -> "Construct a HostSet from an array of "ip:port" strings.
> Add journalnode maintenance node list
> -
>
> Key: HDFS-17223
> URL: https://issues.apache.org/jira/browse/HDFS-17223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: qjm
>Affects Versions: 3.3.6
>Reporter: kuper
>Priority: Major
> Labels: pull-request-available
>
> * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes
> are available and 1 journal node fails to start due to machine issues, it
> will result in a long initialization time for the namenode (around 30-40
> minutes, depending on the IPC timeout and retry policy configuration).
> * The failed journal node cannot recover immediately, but HDFS can still
> function in this situation. In our production environment, we encountered
> this issue and had to reduce the IPC timeout and adjust the retry policy to
> accelerate the namenode initialization and provide services.
> * I'm wondering if it would be possible to have a journal node maintenance
> list to speed up the namenode initialization knowing that one journal node
> cannot provide services in advance?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[ https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17785388#comment-17785388 ] ASF GitHub Bot commented on HDFS-17223: --- gp1314 commented on PR #6183: URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1807571190 Thank you very much for your review, @xinglin . Based on your suggestions, I have made adjustments to the relevant logic and removed the QJPFakedTranslatorPB class. I look forward to your review again. Additionally, the test failure in hadoop.hdfs.TestDFSUtil is related to https://github.com/apache/hadoop/pull/6249. I will rebase once that issue is resolved. > Add journalnode maintenance node list > - > > Key: HDFS-17223 > URL: https://issues.apache.org/jira/browse/HDFS-17223 > Project: Hadoop HDFS > Issue Type: Improvement > Components: qjm >Affects Versions: 3.3.6 >Reporter: kuper >Priority: Major > Labels: pull-request-available > > * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes > are available and 1 journal node fails to start due to machine issues, it > will result in a long initialization time for the namenode (around 30-40 > minutes, depending on the IPC timeout and retry policy configuration). > * The failed journal node cannot recover immediately, but HDFS can still > function in this situation. In our production environment, we encountered > this issue and had to reduce the IPC timeout and adjust the retry policy to > accelerate the namenode initialization and provide services. > * I'm wondering if it would be possible to have a journal node maintenance > list to speed up the namenode initialization knowing that one journal node > cannot provide services in advance? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17784947#comment-17784947
]
ASF GitHub Bot commented on HDFS-17223:
---
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1805774264
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 48s | | trunk passed |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 42m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 27s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 6s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/6/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 10 new + 320 unchanged
- 1 fixed = 330 total (was 321) |
| +1 :green_heart: | mvnsite | 1m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 36s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 30s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 239m 27s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 397m 46s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestDFSUtil |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6183 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux d91fa82d0681 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2ed5c3526085972f266f954b41eaa76d70120973 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubun
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17784547#comment-17784547
]
ASF GitHub Bot commented on HDFS-17223:
---
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1804197786
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 27s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 21s | | trunk passed |
| +1 :green_heart: | compile | 0m 49s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 34s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/5/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 10 new + 247 unchanged
- 0 fixed = 257 total (was 247) |
| +1 :green_heart: | mvnsite | 0m 43s | | the patch passed |
| -1 :x: | javadoc | 0m 36s |
[/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/5/artifact/out/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
|
hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 generated 1 new + 98
unchanged - 1 fixed = 99 total (was 99) |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 194m 23s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 286m 5s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.qjournal.client.TestQuorumJournalManagerUnit |
| | hadoop.hdfs.TestDFSUtil |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6183 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| una
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17784474#comment-17784474
]
ASF GitHub Bot commented on HDFS-17223:
---
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1803928579
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 25s | | trunk passed |
| +1 :green_heart: | compile | 1m 33s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 12s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 3s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/4/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 10 new + 248 unchanged
- 0 fixed = 258 total (was 248) |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| -1 :x: | javadoc | 0m 57s |
[/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/4/artifact/out/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
|
hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 generated 1 new + 98
unchanged - 1 fixed = 99 total (was 99) |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 256m 11s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 415m 8s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.qjournal.client.TestQuorumJournalManager |
| | hadoop.hdfs.qjournal.client.TestQuorumJournalManagerUnit |
| | hadoop.hdfs.TestDFSUtil |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/61
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17784468#comment-17784468
]
ASF GitHub Bot commented on HDFS-17223:
---
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1803910333
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 8s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 13s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 2s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/3/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 11 new + 248 unchanged
- 0 fixed = 259 total (was 248) |
| +1 :green_heart: | mvnsite | 1m 23s | | the patch passed |
| -1 :x: | javadoc | 0m 58s |
[/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/3/artifact/out/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
|
hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
with JDK Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 generated 1 new + 98
unchanged - 1 fixed = 99 total (was 99) |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 258m 9s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 54s | | The patch does not
generate ASF License warnings. |
| | | 414m 53s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.qjournal.client.TestQuorumJournalManager |
| | hadoop.hdfs.TestDFSUtil |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6183 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname |
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17778714#comment-17778714
]
ASF GitHub Bot commented on HDFS-17223:
---
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1775462405
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 46s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| -1 :x: | javadoc | 1m 10s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/2/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 3s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 222 unchanged
- 0 fixed = 225 total (was 222) |
| +1 :green_heart: | mvnsite | 1m 18s | | the patch passed |
| -1 :x: | javadoc | 1m 0s |
[/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/2/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 1m 35s |
[/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/2/artifact/out/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
|
hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05
with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 generated 1 new + 1
unchanged - 0 fixed = 2 total (was 1) |
| +1 :green_heart: | spotbugs | 3m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 42m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 240m 54s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | asflicense | 0m 42s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/2/artifact/out/results-asflicense.txt)
| T
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[ https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17778231#comment-17778231 ] ASF GitHub Bot commented on HDFS-17223: --- xinglin commented on code in PR #6183: URL: https://github.com/apache/hadoop/pull/6183#discussion_r1367803434 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java: ## @@ -146,6 +149,16 @@ public QuorumJournalManager(Configuration conf, this.nameServiceId = nameServiceId; this.loggers = new AsyncLoggerSet(createLoggers(loggerFactory)); +// Check whether the number of jn maintenance lists is valid +String[] skipNodesHostPort = conf.getTrimmedStrings( +DFS_JOURNALNODE_MAINTENANCE_NODES_KEY, DFS_JOURNALNODE_MAINTENANCE_NODES_DEFAULT); + +int quorumThreshold = (this.loggers.size() / 2) + 1; +Preconditions.checkArgument( +(this.loggers.size() - skipNodesHostPort.length) >= quorumThreshold, Review Comment: Instead of creating a QJPFakedTranslatorPB.java, can we just remove these maintenance JNs from `this.loggers `array? We can also make` quorumThreshold` a class member to remember the minimum number of available JNs and the minimum responses we need to commit a write to HDFS. > Add journalnode maintenance node list > - > > Key: HDFS-17223 > URL: https://issues.apache.org/jira/browse/HDFS-17223 > Project: Hadoop HDFS > Issue Type: Improvement > Components: qjm >Affects Versions: 3.3.6 >Reporter: kuper >Priority: Major > Labels: pull-request-available > > * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes > are available and 1 journal node fails to start due to machine issues, it > will result in a long initialization time for the namenode (around 30-40 > minutes, depending on the IPC timeout and retry policy configuration). > * The failed journal node cannot recover immediately, but HDFS can still > function in this situation. In our production environment, we encountered > this issue and had to reduce the IPC timeout and adjust the retry policy to > accelerate the namenode initialization and provide services. > * I'm wondering if it would be possible to have a journal node maintenance > list to speed up the namenode initialization knowing that one journal node > cannot provide services in advance? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17778215#comment-17778215
]
ASF GitHub Bot commented on HDFS-17223:
---
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1773915010
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 48s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| -1 :x: | javadoc | 1m 38s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6183/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-hdfs in trunk failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 1s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 0s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6183/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 222 unchanged
- 0 fixed = 227 total (was 222) |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| -1 :x: | javadoc | 0m 57s |
[/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6183/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 1m 33s |
[/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6183/1/artifact/out/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
|
hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05
with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 generated 1 new + 1
unchanged - 0 fixed = 2 total (was 1) |
| +1 :green_heart: | spotbugs | 3m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 222m 41s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6183/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | asflicense | 0m 45s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch-w
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[
https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774981#comment-17774981
]
ASF GitHub Bot commented on HDFS-17223:
---
hadoop-yetus commented on PR #6183:
URL: https://github.com/apache/hadoop/pull/6183#issuecomment-1761793729
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 20m 2s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 3s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 38s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 10m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 20s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 6s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 222 unchanged
- 0 fixed = 226 total (was 222) |
| +1 :green_heart: | mvnsite | 1m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| -1 :x: | javadoc | 1m 32s |
[/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/1/artifact/out/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
|
hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05
with JDK Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 generated 1 new + 1
unchanged - 0 fixed = 2 total (was 1) |
| +1 :green_heart: | spotbugs | 3m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 239m 41s | | hadoop-hdfs in the patch
passed. |
| -1 :x: | asflicense | 0m 41s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/1/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 386m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6183/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6183 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 38a52bbd3a3f 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e2fa1
[jira] [Commented] (HDFS-17223) Add journalnode maintenance node list
[ https://issues.apache.org/jira/browse/HDFS-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774855#comment-17774855 ] ASF GitHub Bot commented on HDFS-17223: --- gp1314 opened a new pull request, #6183: URL: https://github.com/apache/hadoop/pull/6183 ### Description of PR * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes are available and 1 journal node fails to start due to machine issues, it will result in a long initialization time for the namenode (around 30-40 minutes, depending on the IPC timeout and retry policy configuration). * The failed journal node cannot recover immediately, but HDFS can still function in this situation. In our production environment, we encountered this issue and had to reduce the IPC timeout and adjust the retry policy to accelerate the namenode initialization and provide services. * I'm wondering if it would be possible to have a journal node maintenance list to speed up the namenode initialization knowing that one journal node cannot provide services in advance? ### How was this patch tested? unit test. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Add journalnode maintenance node list > - > > Key: HDFS-17223 > URL: https://issues.apache.org/jira/browse/HDFS-17223 > Project: Hadoop HDFS > Issue Type: Improvement > Components: qjm >Affects Versions: 3.3.6 >Reporter: kuper >Priority: Major > > * In the case of configuring 3 journal nodes in HDFS, if only 2 journal nodes > are available and 1 journal node fails to start due to machine issues, it > will result in a long initialization time for the namenode (around 30-40 > minutes, depending on the IPC timeout and retry policy configuration). > * The failed journal node cannot recover immediately, but HDFS can still > function in this situation. In our production environment, we encountered > this issue and had to reduce the IPC timeout and adjust the retry policy to > accelerate the namenode initialization and provide services. > * I'm wondering if it would be possible to have a journal node maintenance > list to speed up the namenode initialization knowing that one journal node > cannot provide services in advance? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
