[jira] [Commented] (HDFS-17282) Reconfig 'SlowIoWarningThreshold' parameters for datanode.
[ https://issues.apache.org/jira/browse/HDFS-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794930#comment-17794930 ] ASF GitHub Bot commented on HDFS-17282: --- huangzhaobo99 opened a new pull request, #6338: URL: https://github.com/apache/hadoop/pull/6338 ### Description of PR 1. The threshold related to SlowNodes is https://issues.apache.org/jira/browse/hdfs-16396 dynamic reconfig is supported in, SlowLogThreshold also needs to support reconfig to easily dynamic adjustment of log output on DN. 2. The cost of rolling up DN increases and requires a longer time. ### How was this patch tested? Add Unit Test. > Reconfig 'SlowIoWarningThreshold' parameters for datanode. > -- > > Key: HDFS-17282 > URL: https://issues.apache.org/jira/browse/HDFS-17282 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: huangzhaobo99 >Assignee: huangzhaobo99 >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17282) Reconfig 'SlowIoWarningThreshold' parameters for datanode.
[ https://issues.apache.org/jira/browse/HDFS-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17282: -- Labels: pull-request-available (was: ) > Reconfig 'SlowIoWarningThreshold' parameters for datanode. > -- > > Key: HDFS-17282 > URL: https://issues.apache.org/jira/browse/HDFS-17282 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: huangzhaobo99 >Assignee: huangzhaobo99 >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-17282) Reconfig 'SlowIoWarningThreshold' parameters for datanode.
huangzhaobo99 created HDFS-17282: Summary: Reconfig 'SlowIoWarningThreshold' parameters for datanode. Key: HDFS-17282 URL: https://issues.apache.org/jira/browse/HDFS-17282 Project: Hadoop HDFS Issue Type: Improvement Reporter: huangzhaobo99 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-17282) Reconfig 'SlowIoWarningThreshold' parameters for datanode.
[ https://issues.apache.org/jira/browse/HDFS-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huangzhaobo99 reassigned HDFS-17282: Assignee: huangzhaobo99 > Reconfig 'SlowIoWarningThreshold' parameters for datanode. > -- > > Key: HDFS-17282 > URL: https://issues.apache.org/jira/browse/HDFS-17282 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: huangzhaobo99 >Assignee: huangzhaobo99 >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17281) Added support of reporting RPC round-trip time at NN.
[ https://issues.apache.org/jira/browse/HDFS-17281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794924#comment-17794924 ] ASF GitHub Bot commented on HDFS-17281: --- xinglin opened a new pull request, #6337: URL: https://github.com/apache/hadoop/pull/6337 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Added support of reporting RPC round-trip time at NN. > - > > Key: HDFS-17281 > URL: https://issues.apache.org/jira/browse/HDFS-17281 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Major > Attachments: Screenshot 2023-10-28 at 10.26.41 PM.png > > > We have come across a few cases where the hdfs clients are reporting very bad > latencies, while we don't see similar trends at NN-side. Instead, from > NN-side, the latency metrics seem normal as usual. I attached a screenshot > which we took during an internal investigation at LinkedIn. What was > happening is a token management service was reporting an average latency of 1 > sec in fetching delegation tokens from our NN but at the NN-side, we did not > see anything abnormal. The recent OverallRpcProcessingTime metric we added in > HDFS-17042 did not seem to be sufficient to identify/signal such cases. > We propose to extend the IPC header in hadoop, to communicate call create > time at client-side to IPC servers, so that for each rpc call, the server can > get its round-trip time. > > *Why is OverallRpcProcessingTime not sufficient?* > OverallRpcProcessingTime captures the time starting from when the reader > thread reads in the call from the socket to when the response is sent back to > the client. As a result, it does not capture the time it takes to transmit > the call from client to the server. Besides, we only have a couple of reader > threads to monitor a large number of open connections. It is possible that > many connections become ready to read at the same time. Then, the reader > thread would need to read each call sequentially, leading to a wait time for > many Rpc Calls. We have also hit the case where the callQueue becomes full > (with a total of 25600 requests) and thus reader threads are blocked to add > new Calls into the callQueue. This would lead to a longer latency for all > connections/calls which are ready and wait to be read by reader threads. > Ideally, we want to measure the time between when a socket/call is ready to > read and when it is actually being read by the reader thread. This would give > us the wait time that a call is taking to be read. However, after some Google > search, we failed to find a way to get this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17281) Added support of reporting RPC round-trip time at NN.
[ https://issues.apache.org/jira/browse/HDFS-17281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17281: -- Labels: pull-request-available (was: ) > Added support of reporting RPC round-trip time at NN. > - > > Key: HDFS-17281 > URL: https://issues.apache.org/jira/browse/HDFS-17281 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Major > Labels: pull-request-available > Attachments: Screenshot 2023-10-28 at 10.26.41 PM.png > > > We have come across a few cases where the hdfs clients are reporting very bad > latencies, while we don't see similar trends at NN-side. Instead, from > NN-side, the latency metrics seem normal as usual. I attached a screenshot > which we took during an internal investigation at LinkedIn. What was > happening is a token management service was reporting an average latency of 1 > sec in fetching delegation tokens from our NN but at the NN-side, we did not > see anything abnormal. The recent OverallRpcProcessingTime metric we added in > HDFS-17042 did not seem to be sufficient to identify/signal such cases. > We propose to extend the IPC header in hadoop, to communicate call create > time at client-side to IPC servers, so that for each rpc call, the server can > get its round-trip time. > > *Why is OverallRpcProcessingTime not sufficient?* > OverallRpcProcessingTime captures the time starting from when the reader > thread reads in the call from the socket to when the response is sent back to > the client. As a result, it does not capture the time it takes to transmit > the call from client to the server. Besides, we only have a couple of reader > threads to monitor a large number of open connections. It is possible that > many connections become ready to read at the same time. Then, the reader > thread would need to read each call sequentially, leading to a wait time for > many Rpc Calls. We have also hit the case where the callQueue becomes full > (with a total of 25600 requests) and thus reader threads are blocked to add > new Calls into the callQueue. This would lead to a longer latency for all > connections/calls which are ready and wait to be read by reader threads. > Ideally, we want to measure the time between when a socket/call is ready to > read and when it is actually being read by the reader thread. This would give > us the wait time that a call is taking to be read. However, after some Google > search, we failed to find a way to get this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17281) Added support of reporting RPC round-trip time at NN.
[ https://issues.apache.org/jira/browse/HDFS-17281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Lin updated HDFS-17281: Description: We have come across a few cases where the hdfs clients are reporting very bad latencies, while we don't see similar trends at NN-side. Instead, from NN-side, the latency metrics seem normal as usual. I attached a screenshot which we took during an internal investigation at LinkedIn. What was happening is a token management service was reporting an average latency of 1 sec in fetching delegation tokens from our NN but at the NN-side, we did not see anything abnormal. The recent OverallRpcProcessingTime metric we added in HDFS-17042 did not seem to be sufficient to identify/signal such cases. We propose to extend the IPC header in hadoop, to communicate call create time at client-side to IPC servers, so that for each rpc call, the server can get its round-trip time. *Why is OverallRpcProcessingTime not sufficient?* OverallRpcProcessingTime captures the time starting from when the reader thread reads in the call from the socket to when the response is sent back to the client. As a result, it does not capture the time it takes to transmit the call from client to the server. Besides, we only have a couple of reader threads to monitor a large number of open connections. It is possible that many connections become ready to read at the same time. Then, the reader thread would need to read each call sequentially, leading to a wait time for many Rpc Calls. We have also hit the case where the callQueue becomes full (with a total of 25600 requests) and thus reader threads are blocked to add new Calls into the callQueue. This would lead to a longer latency for all connections/calls which are ready and wait to be read by reader threads. Ideally, we want to measure the time between when a socket/call is ready to read and when it is actually being read by the reader thread. This would give us the wait time that a call is taking to be read. However, after some Google search, we failed to find a way to get this. was: We have come across a few cases where the hdfs clients are reporting very bad latencies, we don't see similar trends at NN-side. Instead, from NN-side, the latency metrics seem normal as usual. I attached a screenshot which we took during an internal investigation at LinkedIn. What was happening is a token management service was reporting an average latency of 1 sec in fetching delegation tokens from our NN but at the NN-side, we did not see anything abnormal. In HDFS-17042, we added OverallRpcProcessingTime for each RPC method. This metric measured the time from when the reader thread reads in a call from the socket connection to when the response for the call is sent back to the client. This metric is supposed to be a reliable signal for what RPC latency hdfs clients are getting (overallProcessingTime + network transfer latency == client latency). > Added support of reporting RPC round-trip time at NN. > - > > Key: HDFS-17281 > URL: https://issues.apache.org/jira/browse/HDFS-17281 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Major > Attachments: Screenshot 2023-10-28 at 10.26.41 PM.png > > > We have come across a few cases where the hdfs clients are reporting very bad > latencies, while we don't see similar trends at NN-side. Instead, from > NN-side, the latency metrics seem normal as usual. I attached a screenshot > which we took during an internal investigation at LinkedIn. What was > happening is a token management service was reporting an average latency of 1 > sec in fetching delegation tokens from our NN but at the NN-side, we did not > see anything abnormal. The recent OverallRpcProcessingTime metric we added in > HDFS-17042 did not seem to be sufficient to identify/signal such cases. > We propose to extend the IPC header in hadoop, to communicate call create > time at client-side to IPC servers, so that for each rpc call, the server can > get its round-trip time. > > *Why is OverallRpcProcessingTime not sufficient?* > OverallRpcProcessingTime captures the time starting from when the reader > thread reads in the call from the socket to when the response is sent back to > the client. As a result, it does not capture the time it takes to transmit > the call from client to the server. Besides, we only have a couple of reader > threads to monitor a large number of open connections. It is possible that > many connections become ready to read at the same time. Then, the reader > thread would need to read each call sequentially, leading to a wait time for > many Rpc Calls. We have also hit the case
[jira] [Updated] (HDFS-17281) Added support of reporting RPC round-trip time at NN.
[ https://issues.apache.org/jira/browse/HDFS-17281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Lin updated HDFS-17281: Description: We have come across a few cases where the hdfs clients are reporting very bad latencies, we don't see similar trends at NN-side. Instead, from NN-side, the latency metrics seem normal as usual. I attached a screenshot which we took during an internal investigation at LinkedIn. What was happening is a token management service was reporting an average latency of 1 sec in fetching delegation tokens from our NN but at the NN-side, we did not see anything abnormal. In HDFS-17042, we added OverallRpcProcessingTime for each RPC method. This metric measured the time from when the reader thread reads in a call from the socket connection to when the response for the call is sent back to the client. This metric is supposed to be a reliable signal for what RPC latency hdfs clients are getting (overallProcessingTime + network transfer latency == client latency). was: We have come across a few cases where the hdfs clients are reporting very bad latencies, we don't see similar trends at NN-side. Instead, from NN-side, the latency metrics seem normal as usual. In HDFS-17042, we added OverallRpcProcessingTime for each RPC method. This metric measured the time from when the reader thread reads in a call from the socket connection to when the response for the call is sent back to the client. This metric is supposed to be a reliable signal for what RPC latency hdfs clients are getting (overallProcessingTime + network transfer latency == client latency). > Added support of reporting RPC round-trip time at NN. > - > > Key: HDFS-17281 > URL: https://issues.apache.org/jira/browse/HDFS-17281 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Major > Attachments: Screenshot 2023-10-28 at 10.26.41 PM.png > > > We have come across a few cases where the hdfs clients are reporting very bad > latencies, we don't see similar trends at NN-side. Instead, from NN-side, the > latency metrics seem normal as usual. I attached a screenshot which we took > during an internal investigation at LinkedIn. What was happening is a token > management service was reporting an average latency of 1 sec in fetching > delegation tokens from our NN but at the NN-side, we did not see anything > abnormal. > > > > In HDFS-17042, we added OverallRpcProcessingTime for each RPC method. This > metric measured the time from when the reader thread reads in a call from the > socket connection to when the response for the call is sent back to the > client. This metric is supposed to be a reliable signal for what RPC latency > hdfs clients are getting (overallProcessingTime + network transfer latency == > client latency). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17281) Added support of reporting RPC round-trip time at NN.
[ https://issues.apache.org/jira/browse/HDFS-17281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Lin updated HDFS-17281: Attachment: Screenshot 2023-10-28 at 10.26.41 PM.png > Added support of reporting RPC round-trip time at NN. > - > > Key: HDFS-17281 > URL: https://issues.apache.org/jira/browse/HDFS-17281 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Major > Attachments: Screenshot 2023-10-28 at 10.26.41 PM.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17281) Added support of reporting RPC round-trip time at NN.
[ https://issues.apache.org/jira/browse/HDFS-17281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xing Lin updated HDFS-17281: Description: We have come across a few cases where the hdfs clients are reporting very bad latencies, we don't see similar trends at NN-side. Instead, from NN-side, the latency metrics seem normal as usual. In HDFS-17042, we added OverallRpcProcessingTime for each RPC method. This metric measured the time from when the reader thread reads in a call from the socket connection to when the response for the call is sent back to the client. This metric is supposed to be a reliable signal for what RPC latency hdfs clients are getting (overallProcessingTime + network transfer latency == client latency). > Added support of reporting RPC round-trip time at NN. > - > > Key: HDFS-17281 > URL: https://issues.apache.org/jira/browse/HDFS-17281 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Xing Lin >Assignee: Xing Lin >Priority: Major > Attachments: Screenshot 2023-10-28 at 10.26.41 PM.png > > > We have come across a few cases where the hdfs clients are reporting very bad > latencies, we don't see similar trends at NN-side. Instead, from NN-side, the > latency metrics seem normal as usual. > > > > In HDFS-17042, we added OverallRpcProcessingTime for each RPC method. This > metric measured the time from when the reader thread reads in a call from the > socket connection to when the response for the call is sent back to the > client. This metric is supposed to be a reliable signal for what RPC latency > hdfs clients are getting (overallProcessingTime + network transfer latency == > client latency). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-17281) Added support of reporting RPC round-trip time at NN.
Xing Lin created HDFS-17281: --- Summary: Added support of reporting RPC round-trip time at NN. Key: HDFS-17281 URL: https://issues.apache.org/jira/browse/HDFS-17281 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Reporter: Xing Lin Assignee: Xing Lin -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17280) Pipeline recovery should better end block in advance when bytes acked greater than half of blocksize.
[
https://issues.apache.org/jira/browse/HDFS-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794919#comment-17794919
]
ASF GitHub Bot commented on HDFS-17280:
---
hadoop-yetus commented on PR #6336:
URL: https://github.com/apache/hadoop/pull/6336#issuecomment-1848252179
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 51s | | trunk passed |
| +1 :green_heart: | compile | 1m 1s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| +1 :green_heart: | compile | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 0m 45s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 1s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 22s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 146m 32s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6336 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d05cd5af8edd 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 35014b19dc983aea464e61bf72e920af1364954a |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/1/testReport/ |
| Max. process+thread count | 535 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6336/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This
[jira] [Created] (HDFS-17280) Pipeline recovery should better end block in advance when bytes acked greater than half of blocksize.
farmmamba created HDFS-17280: Summary: Pipeline recovery should better end block in advance when bytes acked greater than half of blocksize. Key: HDFS-17280 URL: https://issues.apache.org/jira/browse/HDFS-17280 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Reporter: farmmamba -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17275) We should determine whether the block has been deleted in the block report
[
https://issues.apache.org/jira/browse/HDFS-17275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794904#comment-17794904
]
ASF GitHub Bot commented on HDFS-17275:
---
ThinkerLei commented on code in PR #6335:
URL: https://github.com/apache/hadoop/pull/6335#discussion_r1421226464
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:
##
@@ -3325,8 +3329,8 @@ private BlockInfo processReportedBlock(
// Block is on the NN
LOG.debug("In memory blockUCState = {}", ucState);
-// Ignore replicas already scheduled to be removed from the DN
-if(invalidateBlocks.contains(dn, block)) {
+// Ignore replicas already scheduled to be removed from the DN or had been
deleted
+if (invalidateBlocks.contains(dn, block) || storedBlock.isDeleted()) {
Review Comment:
Is it ok to judge storedBlock.isDeleted only？
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:
##
@@ -3325,8 +3329,8 @@ private BlockInfo processReportedBlock(
// Block is on the NN
LOG.debug("In memory blockUCState = {}", ucState);
-// Ignore replicas already scheduled to be removed from the DN
-if(invalidateBlocks.contains(dn, block)) {
+// Ignore replicas already scheduled to be removed from the DN or had been
deleted
+if (invalidateBlocks.contains(dn, block) || storedBlock.isDeleted()) {
Review Comment:
Is it ok to judge storedBlock.isDeleted only？
> We should determine whether the block has been deleted in the block report
> --
>
> Key: HDFS-17275
> URL: https://issues.apache.org/jira/browse/HDFS-17275
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: lei w
>Priority: Minor
> Labels: pull-request-available
>
> Now, we use asynchronous thread MarkedDeleteBlockScrubber to delete block. In
> block report.,We may do some useless block related calculations when blocks
> haven't been added to invalidateBlocks
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17275) We should determine whether the block has been deleted in the block report
[ https://issues.apache.org/jira/browse/HDFS-17275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17275: -- Labels: pull-request-available (was: ) > We should determine whether the block has been deleted in the block report > -- > > Key: HDFS-17275 > URL: https://issues.apache.org/jira/browse/HDFS-17275 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: lei w >Priority: Minor > Labels: pull-request-available > > Now, we use asynchronous thread MarkedDeleteBlockScrubber to delete block. In > block report.,We may do some useless block related calculations when blocks > haven't been added to invalidateBlocks -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17275) We should determine whether the block has been deleted in the block report
[ https://issues.apache.org/jira/browse/HDFS-17275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794903#comment-17794903 ] ASF GitHub Bot commented on HDFS-17275: --- ThinkerLei opened a new pull request, #6335: URL: https://github.com/apache/hadoop/pull/6335 Now we use async thread MarkedDeleteBlockScrubber to delete blocks. In order to avoid unnecessary calculation operations on deleted blocks during block reporting, we should determine whether the block has been deleted during block reporting. > We should determine whether the block has been deleted in the block report > -- > > Key: HDFS-17275 > URL: https://issues.apache.org/jira/browse/HDFS-17275 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: lei w >Priority: Minor > > Now, we use asynchronous thread MarkedDeleteBlockScrubber to delete block. In > block report.,We may do some useless block related calculations when blocks > haven't been added to invalidateBlocks -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17279) RBF: Fix link to Fedbalance document
[ https://issues.apache.org/jira/browse/HDFS-17279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HDFS-17279. - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Fix link to Fedbalance document > - > > Key: HDFS-17279 > URL: https://issues.apache.org/jira/browse/HDFS-17279 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: screenshot-1.png > > > !screenshot-1.png! > Fix link to Fedbalance document cannot be displayed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17279) RBF: Fix link to Fedbalance document
[ https://issues.apache.org/jira/browse/HDFS-17279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794857#comment-17794857 ] Ayush Saxena commented on HDFS-17279: - Committed to trunk. Thanx [~haiyang Hu] for the contribution & [~elgoiri] for the review!!! > RBF: Fix link to Fedbalance document > - > > Key: HDFS-17279 > URL: https://issues.apache.org/jira/browse/HDFS-17279 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Attachments: screenshot-1.png > > > !screenshot-1.png! > Fix link to Fedbalance document cannot be displayed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17279) RBF: Fix link to Fedbalance document
[ https://issues.apache.org/jira/browse/HDFS-17279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794856#comment-17794856 ] ASF GitHub Bot commented on HDFS-17279: --- ayushtkn merged PR #6333: URL: https://github.com/apache/hadoop/pull/6333 > RBF: Fix link to Fedbalance document > - > > Key: HDFS-17279 > URL: https://issues.apache.org/jira/browse/HDFS-17279 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Attachments: screenshot-1.png > > > !screenshot-1.png! > Fix link to Fedbalance document cannot be displayed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17265) RBF: Throwing an exception prevents the permit from being released when using FairnessPolicyController
[
https://issues.apache.org/jira/browse/HDFS-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-17265:
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
> RBF: Throwing an exception prevents the permit from being released when using
> FairnessPolicyController
> --
>
> Key: HDFS-17265
> URL: https://issues.apache.org/jira/browse/HDFS-17265
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: HDFS-17265.patch
>
>
> *Bug description*
> When the router uses FairnessPolicyController, each time a request is
> processed,
> the permit of the ns corresponding to the request will be obtained first
> {*}(method acquirePermit){*},
> and then the information of namenodes corresponding to the ns will be
> obtained{*}(method getOrderedNamenodes){*}.
> getOrderedNamenodes comes after acquirePermit, so if acquirePermit succeeds
> but getOrderedNamenodes throws an exception, the permit cannot be released.
>
> *How to reproduce*
> Use the original code to run the new unit test
> testReleasedWhenExceptionOccurs in this PR
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17265) RBF: Throwing an exception prevents the permit from being released when using FairnessPolicyController
[
https://issues.apache.org/jira/browse/HDFS-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794826#comment-17794826
]
ASF GitHub Bot commented on HDFS-17265:
---
goiri merged PR #6298:
URL: https://github.com/apache/hadoop/pull/6298
> RBF: Throwing an exception prevents the permit from being released when using
> FairnessPolicyController
> --
>
> Key: HDFS-17265
> URL: https://issues.apache.org/jira/browse/HDFS-17265
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-17265.patch
>
>
> *Bug description*
> When the router uses FairnessPolicyController, each time a request is
> processed,
> the permit of the ns corresponding to the request will be obtained first
> {*}(method acquirePermit){*},
> and then the information of namenodes corresponding to the ns will be
> obtained{*}(method getOrderedNamenodes){*}.
> getOrderedNamenodes comes after acquirePermit, so if acquirePermit succeeds
> but getOrderedNamenodes throws an exception, the permit cannot be released.
>
> *How to reproduce*
> Use the original code to run the new unit test
> testReleasedWhenExceptionOccurs in this PR
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17273) Change the way of computing some local variables duration for better debugging
[
https://issues.apache.org/jira/browse/HDFS-17273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794763#comment-17794763
]
ASF GitHub Bot commented on HDFS-17273:
---
hadoop-yetus commented on PR #6321:
URL: https://github.com/apache/hadoop/pull/6321#issuecomment-1847342311
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 18s | | trunk passed |
| +1 :green_heart: | compile | 0m 59s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 0m 52s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 26s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 130m 51s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6321/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6321 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux bc30970d034b 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 759c5e7cc692eaa8ec16c4801e49097d80deeb36 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6321/5/testReport/ |
| Max. process+thread count | 723 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6321/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This
[jira] [Commented] (HDFS-17273) Change the way of computing some local variables duration for better debugging
[
https://issues.apache.org/jira/browse/HDFS-17273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794620#comment-17794620
]
ASF GitHub Bot commented on HDFS-17273:
---
hadoop-yetus commented on PR #6321:
URL: https://github.com/apache/hadoop/pull/6321#issuecomment-1846904722
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 37s | | trunk passed |
| +1 :green_heart: | compile | 1m 1s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 49s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 13s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| +1 :green_heart: | compile | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 0m 45s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 21s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6321/3/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-project/hadoop-hdfs-client: The patch generated 1 new + 55
unchanged - 0 fixed = 56 total (was 55) |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 26s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 132m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6321/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6321 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 498c79dde59e 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8470a557b62d704d341ad36f213e1e4bb6ee27cc |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6321/3/testReport/ |
| Max. process+thread count | 731 (vs. ulimit of 5500) |
| modules | C:
[jira] [Commented] (HDFS-17273) Change the way of computing some local variables duration for better debugging
[
https://issues.apache.org/jira/browse/HDFS-17273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794608#comment-17794608
]
ASF GitHub Bot commented on HDFS-17273:
---
hadoop-yetus commented on PR #6321:
URL: https://github.com/apache/hadoop/pull/6321#issuecomment-1846853046
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 0s | | trunk passed |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 11s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6321/4/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-project/hadoop-hdfs-client: The patch generated 1 new + 55
unchanged - 0 fixed = 56 total (was 55) |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 56s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 47s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 83m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6321/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6321 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 982a83efa2ba 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 75bd7b0542c95b098ad4ee6dfd2b592180024d56 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6321/4/testReport/ |
| Max. process+thread count | 555 (vs. ulimit of 5500) |
| modules | C:
[jira] [Commented] (HDFS-17273) Change the way of computing some local variables duration for better debugging
[ https://issues.apache.org/jira/browse/HDFS-17273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794576#comment-17794576 ] ASF GitHub Bot commented on HDFS-17273: --- haiyang1987 commented on PR #6321: URL: https://github.com/apache/hadoop/pull/6321#issuecomment-1846735817 LGTM +1 > Change the way of computing some local variables duration for better debugging > -- > > Key: HDFS-17273 > URL: https://issues.apache.org/jira/browse/HDFS-17273 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: farmmamba >Assignee: farmmamba >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17273) Change the way of computing some local variables duration for better debugging
[
https://issues.apache.org/jira/browse/HDFS-17273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794575#comment-17794575
]
ASF GitHub Bot commented on HDFS-17273:
---
hfutatzhanghb commented on code in PR #6321:
URL: https://github.com/apache/hadoop/pull/6321#discussion_r1420073517
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java:
##
@@ -937,13 +935,13 @@ void waitForAckedSeqno(long seqno) throws IOException {
try {
dataQueue.wait(1000); // when we receive an ack, we notify on
long duration = Time.monotonicNowNanos() - begin;
- if (duration > writeTimeout * NANOSECONDS_PER_MILLISECOND) {
+ if (TimeUnit.NANOSECONDS.toMillis(duration) > writeTimeout) {
LOG.error("No ack received, took {}ms (threshold={}ms). "
+ "File being written: {}, block: {}, "
+ "Write pipeline datanodes: {}.",
-duration / NANOSECONDS_PER_MILLISECOND, writeTimeout, src,
block, nodes);
+TimeUnit.NANOSECONDS.toSeconds(duration), writeTimeout,
src, block, nodes);
Review Comment:
Fixed it, thanks sir.
> Change the way of computing some local variables duration for better debugging
> --
>
> Key: HDFS-17273
> URL: https://issues.apache.org/jira/browse/HDFS-17273
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Minor
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17273) Change the way of computing some local variables duration for better debugging
[
https://issues.apache.org/jira/browse/HDFS-17273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794572#comment-17794572
]
ASF GitHub Bot commented on HDFS-17273:
---
haiyang1987 commented on code in PR #6321:
URL: https://github.com/apache/hadoop/pull/6321#discussion_r1420072279
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java:
##
@@ -937,13 +935,13 @@ void waitForAckedSeqno(long seqno) throws IOException {
try {
dataQueue.wait(1000); // when we receive an ack, we notify on
long duration = Time.monotonicNowNanos() - begin;
- if (duration > writeTimeout * NANOSECONDS_PER_MILLISECOND) {
+ if (TimeUnit.NANOSECONDS.toMillis(duration) > writeTimeout) {
LOG.error("No ack received, took {}ms (threshold={}ms). "
+ "File being written: {}, block: {}, "
+ "Write pipeline datanodes: {}.",
-duration / NANOSECONDS_PER_MILLISECOND, writeTimeout, src,
block, nodes);
+TimeUnit.NANOSECONDS.toSeconds(duration), writeTimeout,
src, block, nodes);
Review Comment:
TimeUnit.NANOSECONDS.toMillis(duration) ？
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java:
##
@@ -937,13 +935,13 @@ void waitForAckedSeqno(long seqno) throws IOException {
try {
dataQueue.wait(1000); // when we receive an ack, we notify on
long duration = Time.monotonicNowNanos() - begin;
- if (duration > writeTimeout * NANOSECONDS_PER_MILLISECOND) {
+ if (TimeUnit.NANOSECONDS.toMillis(duration) > writeTimeout) {
LOG.error("No ack received, took {}ms (threshold={}ms). "
+ "File being written: {}, block: {}, "
+ "Write pipeline datanodes: {}.",
-duration / NANOSECONDS_PER_MILLISECOND, writeTimeout, src,
block, nodes);
+TimeUnit.NANOSECONDS.toSeconds(duration), writeTimeout,
src, block, nodes);
Review Comment:
TimeUnit.NANOSECONDS.toMillis(duration) ？
> Change the way of computing some local variables duration for better debugging
> --
>
> Key: HDFS-17273
> URL: https://issues.apache.org/jira/browse/HDFS-17273
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Minor
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17273) Change the way of computing some local variables duration for better debugging
[
https://issues.apache.org/jira/browse/HDFS-17273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17794570#comment-17794570
]
ASF GitHub Bot commented on HDFS-17273:
---
hfutatzhanghb commented on code in PR #6321:
URL: https://github.com/apache/hadoop/pull/6321#discussion_r1420066021
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java:
##
@@ -934,14 +936,14 @@ void waitForAckedSeqno(long seqno) throws IOException {
}
try {
dataQueue.wait(1000); // when we receive an ack, we notify on
- long duration = Time.monotonicNow() - begin;
- if (duration > writeTimeout) {
+ long duration = Time.monotonicNowNanos() - begin;
Review Comment:
@haiyang1987 Sir, Thanks a lot for your reviewing, you are welcome. Very
nice suggestions here, Have updated it.
> Change the way of computing some local variables duration for better debugging
> --
>
> Key: HDFS-17273
> URL: https://issues.apache.org/jira/browse/HDFS-17273
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Minor
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
