[jira] [Commented] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[ https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832566#comment-17832566 ] ASF GitHub Bot commented on HDFS-17444: --- xiaojunxiang2023 closed pull request #6684: HDFS-17444. Add getJournalSyncerStarted jmx metrics, to Indicates whether the JournalSyncer thread has been started. URL: https://github.com/apache/hadoop/pull/6684 > Add getJournalSyncerStatus jmx metrics, to Indicates whether the > JournalSyncer thread for the every namespace has enter working state. > -- > > Key: HDFS-17444 > URL: https://issues.apache.org/jira/browse/HDFS-17444 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.4, 3.3.6 >Reporter: xiaojunxiang >Assignee: xiaojunxiang >Priority: Major > Labels: pull-request-available > Attachments: image-2024-03-31-09-31-18-056.png > > > The JornalNode JVM process is not immediately in a normal state until the > JournalSyncer thread is started. > For some management platforms such as Ambari rolling restart JournalNode, we > need a jmx metric to determine whether the JournalSyncer thread is started > and enter working state for current namespace before restarting the next > JournalNode. Otherwise, restart too quickly and more than half of > JournalNodes will be out of order, causing the NameNode to die. > When i add it , the effect is as follows: > !image-2024-03-31-09-31-18-056.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[ https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832567#comment-17832567 ] ASF GitHub Bot commented on HDFS-17444: --- xiaojunxiang2023 opened a new pull request, #6694: URL: https://github.com/apache/hadoop/pull/6694 The JornalNode JVM process is not immediately in a normal state until the JournalSyncer thread is started. For some management platforms such as Ambari rolling restart JournalNode, we need a jmx metric to determine whether the JournalSyncer thread is started and enter working state for current namespace before restarting the next JournalNode. Otherwise, restart too quickly and more than half of JournalNodes will be out of order, causing the NameNode to die. When i add it , the effect is as follows:  > Add getJournalSyncerStatus jmx metrics, to Indicates whether the > JournalSyncer thread for the every namespace has enter working state. > -- > > Key: HDFS-17444 > URL: https://issues.apache.org/jira/browse/HDFS-17444 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.4, 3.3.6 >Reporter: xiaojunxiang >Assignee: xiaojunxiang >Priority: Major > Labels: pull-request-available > Attachments: image-2024-03-31-09-31-18-056.png > > > The JornalNode JVM process is not immediately in a normal state until the > JournalSyncer thread is started. > For some management platforms such as Ambari rolling restart JournalNode, we > need a jmx metric to determine whether the JournalSyncer thread is started > and enter working state for current namespace before restarting the next > JournalNode. Otherwise, restart too quickly and more than half of > JournalNodes will be out of order, causing the NameNode to die. > When i add it , the effect is as follows: > !image-2024-03-31-09-31-18-056.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17408) Reduce the number of quota calculations in FSDirRenameOp
[ https://issues.apache.org/jira/browse/HDFS-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832577#comment-17832577 ] ASF GitHub Bot commented on HDFS-17408: --- zhangshuyan0 commented on PR #6653: URL: https://github.com/apache/hadoop/pull/6653#issuecomment-2028674317 @ThinkerLei Thanks for your reply. So what you mean is: If src directory does not have any storage policy set (that is, files under src use the default storage policy, ALL_DISK for example) and dst is set to ALL_SSD, then after renaming src to dst, the data blocks will be stored on the SSD. Do I understand correctly? > Reduce the number of quota calculations in FSDirRenameOp > > > Key: HDFS-17408 > URL: https://issues.apache.org/jira/browse/HDFS-17408 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: lei w >Assignee: lei w >Priority: Major > Labels: pull-request-available > > During the execution of the rename operation, we first calculate the quota > for the source INode using verifyQuotaForRename, and at the same time, we > calculate the quota for the target INode. Subsequently, in > RenameOperation#removeSrc, RenameOperation#removeSrc4OldRename, and > RenameOperation#addSourceToDestination, the quota for the source directory is > calculated again. In exceptional cases, RenameOperation#restoreDst and > RenameOperation#restoreSource will also perform quota calculations for the > source and target directories. In fact, many of the quota calculations are > redundant and unnecessary, so we should optimize them away. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[
https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832593#comment-17832593
]
ASF GitHub Bot commented on HDFS-17444:
---
hadoop-yetus commented on PR #6684:
URL: https://github.com/apache/hadoop/pull/6684#issuecomment-2028778585
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 38s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 30s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 198m 26s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6684/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 288m 30s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdmin |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6684/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6684 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b03cbb37f28e 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2efc09ba67e36eddf303e10d0034d2f33c348fd6 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6684/10/testReport/ |
| Max. process+thread count | 4183 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://
[jira] [Commented] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[
https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832607#comment-17832607
]
ASF GitHub Bot commented on HDFS-17444:
---
hadoop-yetus commented on PR #6694:
URL: https://github.com/apache/hadoop/pull/6694#issuecomment-2028829436
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 35s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 277m 7s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 435m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6694 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f39efc4d7bd8 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2efc09ba67e36eddf303e10d0034d2f33c348fd6 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/1/testReport/ |
| Max. process+thread count | 3500 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Add getJournalSyncerStat
[jira] [Commented] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[
https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832662#comment-17832662
]
ASF GitHub Bot commented on HDFS-17444:
---
hadoop-yetus commented on PR #6694:
URL: https://github.com/apache/hadoop/pull/6694#issuecomment-2028919577
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 12s | | trunk passed |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 58s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 3m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 273m 52s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 431m 54s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6694 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b37458ff93e5 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8d0ef9a8fc6f24bd559f5c3e10b0b34aa16ab7c6 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/2/testReport/ |
| Max. process+thread count | 3148 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6694/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Add getJournalSyncerStat
[jira] [Commented] (HDFS-17397) Choose another DN as soon as possible, when encountering network issues
[ https://issues.apache.org/jira/browse/HDFS-17397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832686#comment-17832686 ] ASF GitHub Bot commented on HDFS-17397: --- xleoken commented on PR #6591: URL: https://github.com/apache/hadoop/pull/6591#issuecomment-2028999836 cc @Hexiaoqiao @ZanderXu > Choose another DN as soon as possible, when encountering network issues > --- > > Key: HDFS-17397 > URL: https://issues.apache.org/jira/browse/HDFS-17397 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: xleoken >Priority: Minor > Labels: pull-request-available > Attachments: hadoop.png > > > Choose another DN as soon as possible, when encountering network issues. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[ https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832690#comment-17832690 ] ASF GitHub Bot commented on HDFS-17444: --- xiaojunxiang2023 commented on PR #6694: URL: https://github.com/apache/hadoop/pull/6694#issuecomment-2029037464 @hiwangzhihui, Hi~, please help me review it. > Add getJournalSyncerStatus jmx metrics, to Indicates whether the > JournalSyncer thread for the every namespace has enter working state. > -- > > Key: HDFS-17444 > URL: https://issues.apache.org/jira/browse/HDFS-17444 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.4, 3.3.6 >Reporter: xiaojunxiang >Assignee: xiaojunxiang >Priority: Major > Labels: pull-request-available > Attachments: image-2024-03-31-09-31-18-056.png > > > The JornalNode JVM process is not immediately in a normal state until the > JournalSyncer thread is started. > For some management platforms such as Ambari rolling restart JournalNode, we > need a jmx metric to determine whether the JournalSyncer thread is started > and enter working state for current namespace before restarting the next > JournalNode. Otherwise, restart too quickly and more than half of > JournalNodes will be out of order, causing the NameNode to die. > When i add it , the effect is as follows: > !image-2024-03-31-09-31-18-056.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17446) The DataNode adds a log to report its progress during addToReplicasMap execution.
[ https://issues.apache.org/jira/browse/HDFS-17446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832691#comment-17832691 ] ASF GitHub Bot commented on HDFS-17446: --- xiaojunxiang2023 commented on PR #6685: URL: https://github.com/apache/hadoop/pull/6685#issuecomment-2029037794 @hiwangzhihui, Hi~, please help me review it. > The DataNode adds a log to report its progress during addToReplicasMap > execution. > - > > Key: HDFS-17446 > URL: https://issues.apache.org/jira/browse/HDFS-17446 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: xiaojunxiang >Priority: Critical > Labels: pull-request-available > Attachments: image-2024-03-28-23-53-29-057.png > > > Datanodes do not print logs when addToReplicasMap is just started, so we will > not be aware of its progress, or even wait up to 4 hours for a large cluster. > This is very confusing. > A cluster is as follows: > !image-2024-03-28-23-53-29-057.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[
https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832697#comment-17832697
]
ASF GitHub Bot commented on HDFS-17444:
---
hiwangzhihui commented on code in PR #6694:
URL: https://github.com/apache/hadoop/pull/6694#discussion_r1545917921
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNode.java:
##
@@ -40,30 +33,27 @@
import org.apache.hadoop.metrics2.util.MBeans;
import org.apache.hadoop.net.NetUtils;
import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.thirdparty.com.google.common.base.Strings;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
import org.apache.hadoop.tracing.TraceUtils;
-import org.apache.hadoop.util.DiskChecker;
-
-import static
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_JOURNALNODE_HTTP_BIND_HOST_KEY;
-import static org.apache.hadoop.util.ExitUtil.terminate;
-import static org.apache.hadoop.util.Time.now;
-import org.apache.hadoop.util.StringUtils;
-import org.apache.hadoop.util.Tool;
-import org.apache.hadoop.util.ToolRunner;
import org.apache.hadoop.tracing.Tracer;
+import org.apache.hadoop.util.*;
import org.eclipse.jetty.util.ajax.JSON;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import javax.management.ObjectName;
import java.io.File;
import java.io.FileFilter;
import java.io.IOException;
import java.net.InetSocketAddress;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
Review Comment:
Import shouldn't using the "*" symbol
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNode.java:
##
@@ -405,6 +395,18 @@ public boolean accept(File file) {
return JSON.toString(status);
}
+ @Override // JournalNodeMXBean
+ public String getJournalSyncerStatus() {
Review Comment:
It would be more appropriate to return an array of string
> Add getJournalSyncerStatus jmx metrics, to Indicates whether the
> JournalSyncer thread for the every namespace has enter working state.
> --
>
> Key: HDFS-17444
> URL: https://issues.apache.org/jira/browse/HDFS-17444
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.4, 3.3.6
>Reporter: xiaojunxiang
>Assignee: xiaojunxiang
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-03-31-09-31-18-056.png
>
>
> The JornalNode JVM process is not immediately in a normal state until the
> JournalSyncer thread is started.
> For some management platforms such as Ambari rolling restart JournalNode, we
> need a jmx metric to determine whether the JournalSyncer thread is started
> and enter working state for current namespace before restarting the next
> JournalNode. Otherwise, restart too quickly and more than half of
> JournalNodes will be out of order, causing the NameNode to die.
> When i add it , the effect is as follows:
> !image-2024-03-31-09-31-18-056.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[ https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832698#comment-17832698 ] ASF GitHub Bot commented on HDFS-17444: --- xiaojunxiang2023 commented on code in PR #6694: URL: https://github.com/apache/hadoop/pull/6694#discussion_r1545922274 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNode.java: ## @@ -40,30 +33,27 @@ import org.apache.hadoop.metrics2.util.MBeans; import org.apache.hadoop.net.NetUtils; import org.apache.hadoop.security.SecurityUtil; +import org.apache.hadoop.thirdparty.com.google.common.base.Strings; +import org.apache.hadoop.thirdparty.com.google.common.collect.Maps; import org.apache.hadoop.tracing.TraceUtils; -import org.apache.hadoop.util.DiskChecker; - -import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_JOURNALNODE_HTTP_BIND_HOST_KEY; -import static org.apache.hadoop.util.ExitUtil.terminate; -import static org.apache.hadoop.util.Time.now; -import org.apache.hadoop.util.StringUtils; -import org.apache.hadoop.util.Tool; -import org.apache.hadoop.util.ToolRunner; import org.apache.hadoop.tracing.Tracer; +import org.apache.hadoop.util.*; import org.eclipse.jetty.util.ajax.JSON; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import javax.management.ObjectName; import java.io.File; import java.io.FileFilter; import java.io.IOException; import java.net.InetSocketAddress; -import java.util.ArrayList; -import java.util.Collection; -import java.util.HashMap; -import java.util.List; -import java.util.Map; +import java.util.*; Review Comment: ok, I will correct it, thanks > Add getJournalSyncerStatus jmx metrics, to Indicates whether the > JournalSyncer thread for the every namespace has enter working state. > -- > > Key: HDFS-17444 > URL: https://issues.apache.org/jira/browse/HDFS-17444 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.4, 3.3.6 >Reporter: xiaojunxiang >Assignee: xiaojunxiang >Priority: Major > Labels: pull-request-available > Attachments: image-2024-03-31-09-31-18-056.png > > > The JornalNode JVM process is not immediately in a normal state until the > JournalSyncer thread is started. > For some management platforms such as Ambari rolling restart JournalNode, we > need a jmx metric to determine whether the JournalSyncer thread is started > and enter working state for current namespace before restarting the next > JournalNode. Otherwise, restart too quickly and more than half of > JournalNodes will be out of order, causing the NameNode to die. > When i add it , the effect is as follows: > !image-2024-03-31-09-31-18-056.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[ https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaojunxiang updated HDFS-17444: Attachment: image-2024-04-01-12-15-42-743.png > Add getJournalSyncerStatus jmx metrics, to Indicates whether the > JournalSyncer thread for the every namespace has enter working state. > -- > > Key: HDFS-17444 > URL: https://issues.apache.org/jira/browse/HDFS-17444 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.4, 3.3.6 >Reporter: xiaojunxiang >Assignee: xiaojunxiang >Priority: Major > Labels: pull-request-available > Attachments: image-2024-03-31-09-31-18-056.png, > image-2024-04-01-12-15-42-743.png > > > The JornalNode JVM process is not immediately in a normal state until the > JournalSyncer thread is started. > For some management platforms such as Ambari rolling restart JournalNode, we > need a jmx metric to determine whether the JournalSyncer thread is started > and enter working state for current namespace before restarting the next > JournalNode. Otherwise, restart too quickly and more than half of > JournalNodes will be out of order, causing the NameNode to die. > When i add it , the effect is as follows: > !image-2024-03-31-09-31-18-056.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[ https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaojunxiang updated HDFS-17444: Description: The JornalNode JVM process is not immediately in a normal state until the JournalSyncer thread is started. For some management platforms such as Ambari rolling restart JournalNode, we need a jmx metric to determine whether the JournalSyncer thread is started and enter working state for current namespace before restarting the next JournalNode. Otherwise, restart too quickly and more than half of JournalNodes will be out of order, causing the NameNode to die. When i add it , the effect is as follows: !image-2024-04-01-12-15-42-743.png! was: The JornalNode JVM process is not immediately in a normal state until the JournalSyncer thread is started. For some management platforms such as Ambari rolling restart JournalNode, we need a jmx metric to determine whether the JournalSyncer thread is started and enter working state for current namespace before restarting the next JournalNode. Otherwise, restart too quickly and more than half of JournalNodes will be out of order, causing the NameNode to die. When i add it , the effect is as follows: !image-2024-03-31-09-31-18-056.png! > Add getJournalSyncerStatus jmx metrics, to Indicates whether the > JournalSyncer thread for the every namespace has enter working state. > -- > > Key: HDFS-17444 > URL: https://issues.apache.org/jira/browse/HDFS-17444 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.4, 3.3.6 >Reporter: xiaojunxiang >Assignee: xiaojunxiang >Priority: Major > Labels: pull-request-available > Attachments: image-2024-03-31-09-31-18-056.png, > image-2024-04-01-12-15-42-743.png > > > The JornalNode JVM process is not immediately in a normal state until the > JournalSyncer thread is started. > For some management platforms such as Ambari rolling restart JournalNode, we > need a jmx metric to determine whether the JournalSyncer thread is started > and enter working state for current namespace before restarting the next > JournalNode. Otherwise, restart too quickly and more than half of > JournalNodes will be out of order, causing the NameNode to die. > When i add it , the effect is as follows: > !image-2024-04-01-12-15-42-743.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[ https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaojunxiang updated HDFS-17444: Attachment: (was: image-2024-03-31-09-31-18-056.png) > Add getJournalSyncerStatus jmx metrics, to Indicates whether the > JournalSyncer thread for the every namespace has enter working state. > -- > > Key: HDFS-17444 > URL: https://issues.apache.org/jira/browse/HDFS-17444 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.4, 3.3.6 >Reporter: xiaojunxiang >Assignee: xiaojunxiang >Priority: Major > Labels: pull-request-available > Attachments: image-2024-04-01-12-16-26-932.png > > > The JornalNode JVM process is not immediately in a normal state until the > JournalSyncer thread is started. > For some management platforms such as Ambari rolling restart JournalNode, we > need a jmx metric to determine whether the JournalSyncer thread is started > and enter working state for current namespace before restarting the next > JournalNode. Otherwise, restart too quickly and more than half of > JournalNodes will be out of order, causing the NameNode to die. > When i add it , the effect is as follows: > !image-2024-04-01-12-15-42-743.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[ https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaojunxiang updated HDFS-17444: Attachment: (was: image-2024-04-01-12-15-42-743.png) > Add getJournalSyncerStatus jmx metrics, to Indicates whether the > JournalSyncer thread for the every namespace has enter working state. > -- > > Key: HDFS-17444 > URL: https://issues.apache.org/jira/browse/HDFS-17444 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.4, 3.3.6 >Reporter: xiaojunxiang >Assignee: xiaojunxiang >Priority: Major > Labels: pull-request-available > Attachments: image-2024-04-01-12-16-26-932.png > > > The JornalNode JVM process is not immediately in a normal state until the > JournalSyncer thread is started. > For some management platforms such as Ambari rolling restart JournalNode, we > need a jmx metric to determine whether the JournalSyncer thread is started > and enter working state for current namespace before restarting the next > JournalNode. Otherwise, restart too quickly and more than half of > JournalNodes will be out of order, causing the NameNode to die. > When i add it , the effect is as follows: > !image-2024-04-01-12-15-42-743.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[ https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaojunxiang updated HDFS-17444: Description: The JornalNode JVM process is not immediately in a normal state until the JournalSyncer thread is started. For some management platforms such as Ambari rolling restart JournalNode, we need a jmx metric to determine whether the JournalSyncer thread is started and enter working state for current namespace before restarting the next JournalNode. Otherwise, restart too quickly and more than half of JournalNodes will be out of order, causing the NameNode to die. When i add it , the effect is as follows: !image-2024-04-01-12-16-26-932.png! was: The JornalNode JVM process is not immediately in a normal state until the JournalSyncer thread is started. For some management platforms such as Ambari rolling restart JournalNode, we need a jmx metric to determine whether the JournalSyncer thread is started and enter working state for current namespace before restarting the next JournalNode. Otherwise, restart too quickly and more than half of JournalNodes will be out of order, causing the NameNode to die. When i add it , the effect is as follows: !image-2024-04-01-12-15-42-743.png! > Add getJournalSyncerStatus jmx metrics, to Indicates whether the > JournalSyncer thread for the every namespace has enter working state. > -- > > Key: HDFS-17444 > URL: https://issues.apache.org/jira/browse/HDFS-17444 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.4, 3.3.6 >Reporter: xiaojunxiang >Assignee: xiaojunxiang >Priority: Major > Labels: pull-request-available > Attachments: image-2024-04-01-12-16-26-932.png > > > The JornalNode JVM process is not immediately in a normal state until the > JournalSyncer thread is started. > For some management platforms such as Ambari rolling restart JournalNode, we > need a jmx metric to determine whether the JournalSyncer thread is started > and enter working state for current namespace before restarting the next > JournalNode. Otherwise, restart too quickly and more than half of > JournalNodes will be out of order, causing the NameNode to die. > When i add it , the effect is as follows: > !image-2024-04-01-12-16-26-932.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17444) Add getJournalSyncerStatus jmx metrics, to Indicates whether the JournalSyncer thread for the every namespace has enter working state.
[ https://issues.apache.org/jira/browse/HDFS-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaojunxiang updated HDFS-17444: Attachment: image-2024-04-01-12-16-26-932.png > Add getJournalSyncerStatus jmx metrics, to Indicates whether the > JournalSyncer thread for the every namespace has enter working state. > -- > > Key: HDFS-17444 > URL: https://issues.apache.org/jira/browse/HDFS-17444 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.4, 3.3.6 >Reporter: xiaojunxiang >Assignee: xiaojunxiang >Priority: Major > Labels: pull-request-available > Attachments: image-2024-04-01-12-16-26-932.png > > > The JornalNode JVM process is not immediately in a normal state until the > JournalSyncer thread is started. > For some management platforms such as Ambari rolling restart JournalNode, we > need a jmx metric to determine whether the JournalSyncer thread is started > and enter working state for current namespace before restarting the next > JournalNode. Otherwise, restart too quickly and more than half of > JournalNodes will be out of order, causing the NameNode to die. > When i add it , the effect is as follows: > !image-2024-04-01-12-15-42-743.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
