[jira] [Commented] (HDFS-17242) Make congestion backoff time configurable
[
https://issues.apache.org/jira/browse/HDFS-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796056#comment-17796056
]
ASF GitHub Bot commented on HDFS-17242:
---
hadoop-yetus commented on PR #6227:
URL: https://github.com/apache/hadoop/pull/6227#issuecomment-1853456445
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 2s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 18s | | trunk passed |
| +1 :green_heart: | compile | 2m 54s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 2m 50s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 6s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 4s | | the patch passed |
| +1 :green_heart: | compile | 2m 48s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 2m 48s | | the patch passed |
| +1 :green_heart: | compile | 2m 44s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 2m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 37s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 5s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 11s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 50s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 185m 53s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6227/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 291m 3s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6227/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6227 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 24026eab9525 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / efefa15c13b687d6bf3a07978486ef6bdcbdf432 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-
[jira] [Commented] (HDFS-17242) Make congestion backoff time configurable
[ https://issues.apache.org/jira/browse/HDFS-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796076#comment-17796076 ] ASF GitHub Bot commented on HDFS-17242: --- hfutatzhanghb commented on PR #6227: URL: https://github.com/apache/hadoop/pull/6227#issuecomment-1853468584 @ayushtkn @tomscut Hi, sir. The failed UT was not related with this PR. > Make congestion backoff time configurable > - > > Key: HDFS-17242 > URL: https://issues.apache.org/jira/browse/HDFS-17242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > > Currently，if we enable congestion backoff, we will actually invoke > backOffIfNecessary method. and the backoff time is computed using > CONGESTION_BACKOFF_MEAN_TIME_IN_MS and CONGESTION_BACK_OFF_MAX_TIME_IN_MS > which are hard-code. We should better make them configurable to compute > backoff sleep time flexibly. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17280) Pipeline recovery should better end block in advance when bytes acked greater than half of blocksize.
[
https://issues.apache.org/jira/browse/HDFS-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796162#comment-17796162
]
ASF GitHub Bot commented on HDFS-17280:
---
hadoop-yetus commented on PR #6336:
URL: https://github.com/apache/hadoop/pull/6336#issuecomment-1853712975
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 28s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 21s | | trunk passed |
| +1 :green_heart: | compile | 6m 3s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 5m 53s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 51s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 19s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 36s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 2s | | the patch passed |
| +1 :green_heart: | compile | 6m 2s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 6m 2s | | the patch passed |
| +1 :green_heart: | compile | 6m 12s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 6m 12s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/14/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 24s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/14/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 1 new + 93 unchanged - 0 fixed =
94 total (was 93) |
| +1 :green_heart: | mvnsite | 2m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 32s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 3s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 6m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 22s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 261m 27s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not
generate ASF License warnings. |
| | | 451m 32s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/14/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6336 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ba3a86c6f838 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c6b625b0727603bd62a22cb0530aec4b13131846 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib
[jira] [Commented] (HDFS-17242) Make congestion backoff time configurable
[ https://issues.apache.org/jira/browse/HDFS-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796264#comment-17796264 ] ASF GitHub Bot commented on HDFS-17242: --- tomscut merged PR #6227: URL: https://github.com/apache/hadoop/pull/6227 > Make congestion backoff time configurable > - > > Key: HDFS-17242 > URL: https://issues.apache.org/jira/browse/HDFS-17242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > > Currently，if we enable congestion backoff, we will actually invoke > backOffIfNecessary method. and the backoff time is computed using > CONGESTION_BACKOFF_MEAN_TIME_IN_MS and CONGESTION_BACK_OFF_MAX_TIME_IN_MS > which are hard-code. We should better make them configurable to compute > backoff sleep time flexibly. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17242) Make congestion backoff time configurable
[ https://issues.apache.org/jira/browse/HDFS-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796266#comment-17796266 ] ASF GitHub Bot commented on HDFS-17242: --- tomscut commented on PR #6227: URL: https://github.com/apache/hadoop/pull/6227#issuecomment-1853870885 Thanks @hfutatzhanghb for your contribution! Thanks @ayushtkn and @xinglin for the review! > Make congestion backoff time configurable > - > > Key: HDFS-17242 > URL: https://issues.apache.org/jira/browse/HDFS-17242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > > Currently，if we enable congestion backoff, we will actually invoke > backOffIfNecessary method. and the backoff time is computed using > CONGESTION_BACKOFF_MEAN_TIME_IN_MS and CONGESTION_BACK_OFF_MAX_TIME_IN_MS > which are hard-code. We should better make them configurable to compute > backoff sleep time flexibly. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17242) Make congestion backoff time configurable
[ https://issues.apache.org/jira/browse/HDFS-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li resolved HDFS-17242. --- Fix Version/s: 3.4.0 Resolution: Fixed > Make congestion backoff time configurable > - > > Key: HDFS-17242 > URL: https://issues.apache.org/jira/browse/HDFS-17242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Currently，if we enable congestion backoff, we will actually invoke > backOffIfNecessary method. and the backoff time is computed using > CONGESTION_BACKOFF_MEAN_TIME_IN_MS and CONGESTION_BACK_OFF_MAX_TIME_IN_MS > which are hard-code. We should better make them configurable to compute > backoff sleep time flexibly. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17242) Make congestion backoff time configurable
[ https://issues.apache.org/jira/browse/HDFS-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796270#comment-17796270 ] ASF GitHub Bot commented on HDFS-17242: --- hfutatzhanghb commented on PR #6227: URL: https://github.com/apache/hadoop/pull/6227#issuecomment-1853901143 Thanks @tomscut for helping review and merge. Thanks @ayushtkn @xinglin for the review!!! > Make congestion backoff time configurable > - > > Key: HDFS-17242 > URL: https://issues.apache.org/jira/browse/HDFS-17242 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.4.0 >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Currently，if we enable congestion backoff, we will actually invoke > backOffIfNecessary method. and the backoff time is computed using > CONGESTION_BACKOFF_MEAN_TIME_IN_MS and CONGESTION_BACK_OFF_MAX_TIME_IN_MS > which are hard-code. We should better make them configurable to compute > backoff sleep time flexibly. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17282) Reconfig 'SlowIoWarningThreshold' parameters for datanode.
[
https://issues.apache.org/jira/browse/HDFS-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796319#comment-17796319
]
ASF GitHub Bot commented on HDFS-17282:
---
hadoop-yetus commented on PR #6338:
URL: https://github.com/apache/hadoop/pull/6338#issuecomment-1854057259
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 58s | | trunk passed |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 2s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 110 unchanged - 1
fixed = 110 total (was 111) |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 56s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 251m 23s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6338/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 406m 10s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6338/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6338 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9875ddf1dc0b 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1afb4e31b6dc9c8d837205cca741e018325de4f7 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6338/3/testReport/ |
| Max. process+thread count | 2107 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdf
[jira] [Commented] (HDFS-17284) Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks during block recovery
[ https://issues.apache.org/jira/browse/HDFS-17284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796346#comment-17796346 ] ASF GitHub Bot commented on HDFS-17284: --- tasanuma commented on PR #6348: URL: https://github.com/apache/hadoop/pull/6348#issuecomment-1854136818 Thanks for reporting the issue and submitting the PR, @zhtttylz. The main fix looks good. But I have concerns about the unit test because they pass even without the main fix. Also, I don't think `...Fixed` or `...Origin` are good names for methods. Source code changes all the time, so method names should reflect what they do. I think you can create an overflow situation by passing large numbers as arguments into the `verifyPendingRecoveryTasks` method. `TestDatanodeManager#testPendingRecoveryTasks` might give you some ideas. > Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks > during block recovery > -- > > Key: HDFS-17284 > URL: https://issues.apache.org/jira/browse/HDFS-17284 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > > Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks > during block recovery -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17284) Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks during block recovery
[ https://issues.apache.org/jira/browse/HDFS-17284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796347#comment-17796347 ] ASF GitHub Bot commented on HDFS-17284: --- tasanuma commented on code in PR #6348: URL: https://github.com/apache/hadoop/pull/6348#discussion_r1425516248 ## hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java: ## @@ -36,6 +36,7 @@ import java.util.Random; import java.util.Set; +import org.apache.hadoop.hdfs.TestLeaseRecoveryStriped; Review Comment: This seems not to be used. > Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks > during block recovery > -- > > Key: HDFS-17284 > URL: https://issues.apache.org/jira/browse/HDFS-17284 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > > Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks > during block recovery -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17042) Add rpcCallSuccesses and OverallRpcProcessingTime to RpcMetrics for Namenode
[
https://issues.apache.org/jira/browse/HDFS-17042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-17042:
Fix Version/s: 3.3.9
> Add rpcCallSuccesses and OverallRpcProcessingTime to RpcMetrics for Namenode
>
>
> Key: HDFS-17042
> URL: https://issues.apache.org/jira/browse/HDFS-17042
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.4.0, 3.3.9
>Reporter: Xing Lin
>Assignee: Xing Lin
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> We'd like to add two new types of metrics to the existing NN
> RpcMetrics/RpcDetailedMetrics. These two metrics can then be used as part of
> SLA/SLO for the HDFS service.
> * {_}RpcCallSuccesses{_}: it measures the number of RPC requests where they
> are successfully processed by a NN (e.g., with a response with an RpcStatus
> {_}RpcStatusProto.SUCCESS){_}{_}.{_} Then, together with {_}RpcQueueNumOps
> ({_}which refers the total number of RPC requests{_}){_}, we can derive the
> RpcErrorRate for our NN, as (RpcQueueNumOps - RpcCallSuccesses) /
> RpcQueueNumOps.
> * OverallRpcProcessingTime for each RPC method: this metric measures the
> overall RPC processing time for each RPC method at the NN. It covers the time
> from when a request arrives at the NN to when a response is sent back. We are
> already emitting processingTime for each RPC method today in
> RpcDetailedMetrics. We want to extend it to emit overallRpcProcessingTime for
> each RPC method, which includes enqueueTime, queueTime, processingTime,
> responseTime, and handlerTime.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17280) Pipeline recovery should better end block in advance when bytes acked greater than half of blocksize.
[
https://issues.apache.org/jira/browse/HDFS-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796362#comment-17796362
]
ASF GitHub Bot commented on HDFS-17280:
---
hadoop-yetus commented on PR #6336:
URL: https://github.com/apache/hadoop/pull/6336#issuecomment-1854218445
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 41s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 37m 2s | | trunk passed |
| +1 :green_heart: | compile | 7m 19s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 6m 42s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 38s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 11s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 36s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 6m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 43m 41s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 12s | | the patch passed |
| +1 :green_heart: | compile | 6m 28s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 6m 28s | | the patch passed |
| +1 :green_heart: | compile | 6m 2s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 6m 2s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/15/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 28s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/15/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 2 new + 93 unchanged - 0 fixed =
95 total (was 93) |
| +1 :green_heart: | mvnsite | 2m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 48s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 13s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 6m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 42m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 23s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 260m 6s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/15/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 463m 19s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/15/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6336 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 7baf8fc218d0 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
[jira] [Commented] (HDFS-17280) Pipeline recovery should better end block in advance when bytes acked greater than half of blocksize.
[
https://issues.apache.org/jira/browse/HDFS-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796365#comment-17796365
]
ASF GitHub Bot commented on HDFS-17280:
---
hadoop-yetus commented on PR #6336:
URL: https://github.com/apache/hadoop/pull/6336#issuecomment-1854239295
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 55s | | trunk passed |
| +1 :green_heart: | compile | 5m 26s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 5m 12s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 16s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 17s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 56s | | the patch passed |
| +1 :green_heart: | compile | 5m 20s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 5m 20s | | the patch passed |
| +1 :green_heart: | compile | 5m 2s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 5m 2s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/16/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 11s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/16/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 2 new + 93 unchanged - 0 fixed =
95 total (was 93) |
| +1 :green_heart: | mvnsite | 2m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 11s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 48s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 31s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 216m 19s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/16/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 387m 56s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6336 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 0c062df8f49f 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool |
[jira] [Commented] (HDFS-17280) Pipeline recovery should better end block in advance when bytes acked greater than half of blocksize.
[
https://issues.apache.org/jira/browse/HDFS-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796457#comment-17796457
]
ASF GitHub Bot commented on HDFS-17280:
---
hadoop-yetus commented on PR #6336:
URL: https://github.com/apache/hadoop/pull/6336#issuecomment-1854694005
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 43s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 39m 8s | | trunk passed |
| +1 :green_heart: | compile | 7m 8s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 7m 3s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 10s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 40s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 6m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 42m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 7s | | the patch passed |
| +1 :green_heart: | compile | 6m 12s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 6m 12s | | the patch passed |
| +1 :green_heart: | compile | 6m 20s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 6m 20s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/17/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 32s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/17/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 1 new + 93 unchanged - 0 fixed =
94 total (was 93) |
| +1 :green_heart: | mvnsite | 2m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 2s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 24s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 6m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 23s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 250m 42s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/17/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 452m 51s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/17/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6336 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e37010472050 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_6
[jira] [Commented] (HDFS-17254) DataNode httpServer has too many worker threads
[
https://issues.apache.org/jira/browse/HDFS-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796481#comment-17796481
]
ASF GitHub Bot commented on HDFS-17254:
---
hadoop-yetus commented on PR #6307:
URL: https://github.com/apache/hadoop/pull/6307#issuecomment-1854788515
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 11m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 42s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 9s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 6s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 270m 23s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6307/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 417m 17s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.server.namenode.TestUpgradeDomainBlockPlacementPolicy |
| | hadoop.hdfs.server.blockmanagement.TestBlockInfoStriped |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.hdfs.server.mover.TestStorageMover |
| | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.mover.TestMover |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6307/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6307 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
m
[jira] [Commented] (HDFS-17254) DataNode httpServer has too many worker threads
[ https://issues.apache.org/jira/browse/HDFS-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796510#comment-17796510 ] ASF GitHub Bot commented on HDFS-17254: --- 2005hithlj commented on PR #6307: URL: https://github.com/apache/hadoop/pull/6307#issuecomment-1854984719 @Hexiaoqiao OK Sir, I have already fixed and committed it. > DataNode httpServer has too many worker threads > --- > > Key: HDFS-17254 > URL: https://issues.apache.org/jira/browse/HDFS-17254 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Liangjun He >Assignee: Liangjun He >Priority: Minor > Labels: pull-request-available > > When optimizing the thread number of high-density storage DN, we found the > number of worker threads for the DataNode httpServer is twice the number of > available cores on node , resulting in too many threads. We can change this > to be configurable. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17254) DataNode httpServer has too many worker threads
[ https://issues.apache.org/jira/browse/HDFS-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796512#comment-17796512 ] ASF GitHub Bot commented on HDFS-17254: --- tomscut commented on code in PR #6307: URL: https://github.com/apache/hadoop/pull/6307#discussion_r1413288934 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java: ## @@ -71,6 +71,8 @@ import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_HTTPS_ADDRESS_KEY; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_HTTP_ADDRESS_KEY; import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_HTTP_INTERNAL_PROXY_PORT; +import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_NETTY_WORKER_NUM_THREADS_KEY; +import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_NETTY_WORKER_NUM_THREADS_DEFAULT; Review Comment: Please add this configuration into `hdfs-default.xml`. Thanks. > DataNode httpServer has too many worker threads > --- > > Key: HDFS-17254 > URL: https://issues.apache.org/jira/browse/HDFS-17254 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Liangjun He >Assignee: Liangjun He >Priority: Minor > Labels: pull-request-available > > When optimizing the thread number of high-density storage DN, we found the > number of worker threads for the DataNode httpServer is twice the number of > available cores on node , resulting in too many threads. We can change this > to be configurable. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-17288) Add a metric to record the number of non-lastBlocks which have non-complete blocksize
farmmamba created HDFS-17288: Summary: Add a metric to record the number of non-lastBlocks which have non-complete blocksize Key: HDFS-17288 URL: https://issues.apache.org/jira/browse/HDFS-17288 Project: Hadoop HDFS Issue Type: Improvement Reporter: farmmamba Add a metric to record the number of non-lastBlocks which have non-complete blocksize. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17254) DataNode httpServer has too many worker threads
[ https://issues.apache.org/jira/browse/HDFS-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796513#comment-17796513 ] ASF GitHub Bot commented on HDFS-17254: --- tomscut commented on code in PR #6307: URL: https://github.com/apache/hadoop/pull/6307#discussion_r1426092305 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java: ## @@ -144,7 +146,9 @@ public DatanodeHttpServer(final Configuration conf, confForCreate.set(FsPermission.UMASK_LABEL, "000"); this.bossGroup = new NioEventLoopGroup(); -this.workerGroup = new NioEventLoopGroup(); +int workerThreads = conf.getInt(DFS_DATANODE_NETTY_WORKER_NUM_THREADS_KEY, +DFS_DATANODE_NETTY_WORKER_NUM_THREADS_DEFAULT); +this.workerGroup = new NioEventLoopGroup(workerThreads); Review Comment: Better to use a cached thread pool here. like `this.workerGroup = new NioEventLoopGroup(workerCount, Executors.newCachedThreadPool())`. > DataNode httpServer has too many worker threads > --- > > Key: HDFS-17254 > URL: https://issues.apache.org/jira/browse/HDFS-17254 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Liangjun He >Assignee: Liangjun He >Priority: Minor > Labels: pull-request-available > > When optimizing the thread number of high-density storage DN, we found the > number of worker threads for the DataNode httpServer is twice the number of > available cores on node , resulting in too many threads. We can change this > to be configurable. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17254) DataNode httpServer has too many worker threads
[ https://issues.apache.org/jira/browse/HDFS-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796514#comment-17796514 ] ASF GitHub Bot commented on HDFS-17254: --- tomscut commented on code in PR #6307: URL: https://github.com/apache/hadoop/pull/6307#discussion_r1426092305 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java: ## @@ -144,7 +146,9 @@ public DatanodeHttpServer(final Configuration conf, confForCreate.set(FsPermission.UMASK_LABEL, "000"); this.bossGroup = new NioEventLoopGroup(); -this.workerGroup = new NioEventLoopGroup(); +int workerThreads = conf.getInt(DFS_DATANODE_NETTY_WORKER_NUM_THREADS_KEY, +DFS_DATANODE_NETTY_WORKER_NUM_THREADS_DEFAULT); +this.workerGroup = new NioEventLoopGroup(workerThreads); Review Comment: Better to use a cached thread pool here. `this.workerGroup = new NioEventLoopGroup(workerCount, Executors.newCachedThreadPool())`. > DataNode httpServer has too many worker threads > --- > > Key: HDFS-17254 > URL: https://issues.apache.org/jira/browse/HDFS-17254 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Liangjun He >Assignee: Liangjun He >Priority: Minor > Labels: pull-request-available > > When optimizing the thread number of high-density storage DN, we found the > number of worker threads for the DataNode httpServer is twice the number of > available cores on node , resulting in too many threads. We can change this > to be configurable. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-17289) Should not consider the size of non-lastBlocks equals to complete block size.
farmmamba created HDFS-17289: Summary: Should not consider the size of non-lastBlocks equals to complete block size. Key: HDFS-17289 URL: https://issues.apache.org/jira/browse/HDFS-17289 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 3.3.6 Reporter: farmmamba -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-17289) Should not consider the size of non-lastBlocks equals to complete block size.
[ https://issues.apache.org/jira/browse/HDFS-17289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] farmmamba reassigned HDFS-17289: Assignee: farmmamba > Should not consider the size of non-lastBlocks equals to complete block size. > - > > Key: HDFS-17289 > URL: https://issues.apache.org/jira/browse/HDFS-17289 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17282) Reconfig 'SlowIoWarningThreshold' parameters for datanode.
[
https://issues.apache.org/jira/browse/HDFS-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796519#comment-17796519
]
ASF GitHub Bot commented on HDFS-17282:
---
huangzhaobo99 commented on code in PR #6338:
URL: https://github.com/apache/hadoop/pull/6338#discussion_r1426121112
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:
##
@@ -1056,6 +1061,25 @@ private String reconfDiskBalancerParameters(String
property, String newVal)
}
}
+ private String reconfSlowIoWarningThresholdParameters(String property,
+ String newVal) throws ReconfigurationException {
Review Comment:
Thanks! I have modified the settings.
> Reconfig 'SlowIoWarningThreshold' parameters for datanode.
> --
>
> Key: HDFS-17282
> URL: https://issues.apache.org/jira/browse/HDFS-17282
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: huangzhaobo99
>Assignee: huangzhaobo99
>Priority: Minor
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17289) Should not consider the size of non-lastBlocks equals to complete block size.
[ https://issues.apache.org/jira/browse/HDFS-17289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17289: -- Labels: pull-request-available (was: ) > Should not consider the size of non-lastBlocks equals to complete block size. > - > > Key: HDFS-17289 > URL: https://issues.apache.org/jira/browse/HDFS-17289 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17289) Should not consider the size of non-lastBlocks equals to complete block size.
[ https://issues.apache.org/jira/browse/HDFS-17289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796530#comment-17796530 ] ASF GitHub Bot commented on HDFS-17289: --- hfutatzhanghb opened a new pull request, #6357: URL: https://github.com/apache/hadoop/pull/6357 ### Description of PR Currently, some codes in HDFS use an implicit condition that all blocks which are not last block in file have full blocksize(default 128MB). This is not advisable, we should change those codes. > Should not consider the size of non-lastBlocks equals to complete block size. > - > > Key: HDFS-17289 > URL: https://issues.apache.org/jira/browse/HDFS-17289 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17284) Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks during block recovery
[ https://issues.apache.org/jira/browse/HDFS-17284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796534#comment-17796534 ] ASF GitHub Bot commented on HDFS-17284: --- zhtttylz commented on code in PR #6348: URL: https://github.com/apache/hadoop/pull/6348#discussion_r1426153084 ## hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java: ## @@ -36,6 +36,7 @@ import java.util.Random; import java.util.Set; +import org.apache.hadoop.hdfs.TestLeaseRecoveryStriped; Review Comment: Thank you for your valuable suggestion. I greatly appreciate it and will promptly make the necessary changes to the code! > Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks > during block recovery > -- > > Key: HDFS-17284 > URL: https://issues.apache.org/jira/browse/HDFS-17284 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Hualong Zhang >Assignee: Hualong Zhang >Priority: Major > Labels: pull-request-available > > Fix int overflow in calculating numEcReplicatedTasks and numReplicationTasks > during block recovery -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17289) Should not consider the size of non-lastBlocks equals to complete block size.
[
https://issues.apache.org/jira/browse/HDFS-17289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796556#comment-17796556
]
ASF GitHub Bot commented on HDFS-17289:
---
hadoop-yetus commented on PR #6357:
URL: https://github.com/apache/hadoop/pull/6357#issuecomment-1855148403
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 37s | | trunk passed |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 26s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 12s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 52s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 91m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6357/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6357 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c5fec03d7ba8 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c10111d7f71eec2e275e1c1ec89e652160c861da |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6357/1/testReport/ |
| Max. process+thread count | 557 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6357/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
[jira] [Commented] (HDFS-17289) Should not consider the size of non-lastBlocks equals to complete block size.
[ https://issues.apache.org/jira/browse/HDFS-17289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796563#comment-17796563 ] ASF GitHub Bot commented on HDFS-17289: --- hfutatzhanghb commented on PR #6357: URL: https://github.com/apache/hadoop/pull/6357#issuecomment-1855178376 @Hexiaoqiao @tomscut @zhangshuyan0 Hi, sir. Could you please help me review this PR when you have free time? Thanks a lot. > Should not consider the size of non-lastBlocks equals to complete block size. > - > > Key: HDFS-17289 > URL: https://issues.apache.org/jira/browse/HDFS-17289 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: farmmamba >Assignee: farmmamba >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17282) Reconfig 'SlowIoWarningThreshold' parameters for datanode.
[ https://issues.apache.org/jira/browse/HDFS-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HDFS-17282. - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Reconfig 'SlowIoWarningThreshold' parameters for datanode. > -- > > Key: HDFS-17282 > URL: https://issues.apache.org/jira/browse/HDFS-17282 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: huangzhaobo99 >Assignee: huangzhaobo99 >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17282) Reconfig 'SlowIoWarningThreshold' parameters for datanode.
[ https://issues.apache.org/jira/browse/HDFS-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796568#comment-17796568 ] ASF GitHub Bot commented on HDFS-17282: --- ayushtkn merged PR #6338: URL: https://github.com/apache/hadoop/pull/6338 > Reconfig 'SlowIoWarningThreshold' parameters for datanode. > -- > > Key: HDFS-17282 > URL: https://issues.apache.org/jira/browse/HDFS-17282 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: huangzhaobo99 >Assignee: huangzhaobo99 >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17282) Reconfig 'SlowIoWarningThreshold' parameters for datanode.
[ https://issues.apache.org/jira/browse/HDFS-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796569#comment-17796569 ] Ayush Saxena commented on HDFS-17282: - Committed to trunk. Thanx [~huangzhaobo99] for the contribution!!! > Reconfig 'SlowIoWarningThreshold' parameters for datanode. > -- > > Key: HDFS-17282 > URL: https://issues.apache.org/jira/browse/HDFS-17282 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: huangzhaobo99 >Assignee: huangzhaobo99 >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16993) Datanode supports configure TopN DatanodeNetworkCounts
[ https://issues.apache.org/jira/browse/HDFS-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796581#comment-17796581 ] ASF GitHub Bot commented on HDFS-16993: --- hfutatzhanghb commented on PR #5597: URL: https://github.com/apache/hadoop/pull/5597#issuecomment-1855260225 @ayushtkn @Hexiaoqiao @tomscut Hi, sir. I think we should better push this PR forward. Because it waste datanode's memory. As the below screeshot show , it has 27K networkError key values!!! https://github.com/apache/hadoop/assets/25115709/d58f6421-ef70-4e45-9f93-ba51518bda31";> > Datanode supports configure TopN DatanodeNetworkCounts > -- > > Key: HDFS-16993 > URL: https://issues.apache.org/jira/browse/HDFS-16993 > Project: Hadoop HDFS > Issue Type: Wish >Affects Versions: 3.3.5 >Reporter: farmmamba >Priority: Major > Labels: pull-request-available > > In our prod environment, we try to collect datanode metrics every 15s through > jmx_exporter. we found the datanodenetworkerror metric generates a lot. > for example, if we have a cluster with 1000 datanodes, every datanode may > generate 999 datanodenetworkerror metrics, and overall datanodes will > generate 1000 multiple 999 = 999000 metrics. This is a very expensive > operation. In most scenarios, we only need the topN of it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16993) Datanode supports configure TopN DatanodeNetworkCounts
[ https://issues.apache.org/jira/browse/HDFS-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17796583#comment-17796583 ] ASF GitHub Bot commented on HDFS-16993: --- hfutatzhanghb commented on PR #5597: URL: https://github.com/apache/hadoop/pull/5597#issuecomment-1855265316 https://github.com/apache/hadoop/assets/25115709/0f35d396-6754-4a3d-b65a-a8d8d9fc0ed8";> almost 60k+ on another datanode. > Datanode supports configure TopN DatanodeNetworkCounts > -- > > Key: HDFS-16993 > URL: https://issues.apache.org/jira/browse/HDFS-16993 > Project: Hadoop HDFS > Issue Type: Wish >Affects Versions: 3.3.5 >Reporter: farmmamba >Priority: Major > Labels: pull-request-available > > In our prod environment, we try to collect datanode metrics every 15s through > jmx_exporter. we found the datanodenetworkerror metric generates a lot. > for example, if we have a cluster with 1000 datanodes, every datanode may > generate 999 datanodenetworkerror metrics, and overall datanodes will > generate 1000 multiple 999 = 999000 metrics. This is a very expensive > operation. In most scenarios, we only need the topN of it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
