[jira] [Updated] (HDFS-16614) Improve balancer operation strategy and performance
[ https://issues.apache.org/jira/browse/HDFS-16614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangHua Zhu updated HDFS-16614: Description: When the Balancer program is run, it does some work in the following order: 1. Obtain available datanode information from NameNode. 2. Classify and calculate the average utilization according to StorageType. Here, some sets will be obtained in combination with the set thresholds: overUtilized, aboveAvgUtilized, belowAvgUtilized, and underUtilized. 3. According to some calculations, the source and target related to the transfer data are obtained. The source is used for the source end, and the target is used for the data receiving end. 4. Start the data transfer work in parallel. In this process, run iteratively. In this process, the threshold is unified and applied to all StorageTypes, which seems to be a bit rough, because one of the StorageTypes cannot be distinguished, which is based on the currently supported heterogeneous storage. There is an online cluster with more than 2000 nodes, and there is an imbalance in node storage. E.g: !image-2022-06-02-13-18-33-213.png! Here, the average utilization of the cluster is 78%, but the utilization of most nodes is between 85% and 90%. When the balancer is turned on, we find that 85% of the nodes are working as sources. In this case, we think it is not reasonable, because it will occupy more network resources in the cluster, and it will be beneficial to the normal work of the cluster to do some effective restrictions. So here are some changes to make: 1. When the balancer is running, we should actively prompt the suggested value of the threshold related to StorageType. For example: [[DISK, 10%], [SSD, 8%]...] 2. Support to set threshold according to StorageType and work. 3. Add an option to prohibit nodes below the threshold from joining the Source set. This is to allow nodes with high utilization to transfer data as soon as possible, which is good for balance. 4. Add new support. If there are a lot of datanode usage in the cluster, it should remain unchanged. For example, the utilization rate of 40% of the nodes in the cluster is 75% to 80%, and these nodes should not join the Source set. Of course this support needs to be specified by the user at runtime. was: When the Balancer program is run, it does some work in the following order: 1. Obtain available datanode information from NameNode. 2. Classify and calculate the average utilization according to StorageType. Here, some sets will be obtained in combination with the set thresholds: overUtilized, aboveAvgUtilized, belowAvgUtilized, and underUtilized. 3. According to some calculations, the source and target related to the transfer data are obtained. The source is used for the source end, and the target is used for the data receiving end. 4. Start the data transfer work in parallel. In this process, run iteratively. In this process, the threshold is unified and applied to all StorageTypes, which seems to be a bit rough, because one of the StorageTypes cannot be distinguished, which is based on the currently supported heterogeneous storage. There is an online cluster with more than 2000 nodes, and there is an imbalance in node storage. E.g: !image-2022-06-02-13-18-33-213.png! Here, the average utilization of the cluster is 78%, but the utilization of most nodes is between 85% and 90%. When the balancer is turned on, we find that 85% of the nodes are working as sources. In this case, we think it is not reasonable, because it will occupy more network resources in the cluster, and it will be beneficial to the normal work of the cluster to do some effective restrictions. So here are some changes to make: 1. When the balancer is running, it should try to prompt the threshold related to StorageType. For example [[DISK, 10%], [SSD, 8%]...] 2. Support to set threshold according to StorageType and work. 3. Add an option to prohibit nodes below the threshold from joining the Source set. This is to allow nodes with high utilization to transfer data as soon as possible, which is good for balance. 4. Add new support. If there are a lot of datanode usage in the cluster, it should remain unchanged. For example, the utilization rate of 40% of the nodes in the cluster is 75% to 80%, and these nodes should not join the Source set. Of course this support needs to be specified by the user at runtime. > Improve balancer operation strategy and performance > --- > > Key: HDFS-16614 > URL: https://issues.apache.org/jira/browse/HDFS-16614 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover, namenode >Affects Versions: 2.9.2 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Att
[jira] [Updated] (HDFS-16614) Improve balancer operation strategy and performance
[ https://issues.apache.org/jira/browse/HDFS-16614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangHua Zhu updated HDFS-16614: Affects Version/s: 2.9.2 (was: 3.3.0) > Improve balancer operation strategy and performance > --- > > Key: HDFS-16614 > URL: https://issues.apache.org/jira/browse/HDFS-16614 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover, namenode >Affects Versions: 2.9.2 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Attachments: image-2022-06-02-13-18-33-213.png > > > When the Balancer program is run, it does some work in the following order: > 1. Obtain available datanode information from NameNode. > 2. Classify and calculate the average utilization according to StorageType. > Here, some sets will be obtained in combination with the set thresholds: > overUtilized, aboveAvgUtilized, belowAvgUtilized, and underUtilized. > 3. According to some calculations, the source and target related to the > transfer data are obtained. The source is used for the source end, and the > target is used for the data receiving end. > 4. Start the data transfer work in parallel. > In this process, run iteratively. In this process, the threshold is unified > and applied to all StorageTypes, which seems to be a bit rough, because one > of the StorageTypes cannot be distinguished, which is based on the currently > supported heterogeneous storage. > There is an online cluster with more than 2000 nodes, and there is an > imbalance in node storage. E.g: > !image-2022-06-02-13-18-33-213.png! > Here, the average utilization of the cluster is 78%, but the utilization of > most nodes is between 85% and 90%. When the balancer is turned on, we find > that 85% of the nodes are working as sources. In this case, we think it is > not reasonable, because it will occupy more network resources in the cluster, > and it will be beneficial to the normal work of the cluster to do some > effective restrictions. > So here are some changes to make: > 1. When the balancer is running, it should try to prompt the threshold > related to StorageType. For example [[DISK, 10%], [SSD, 8%]...] > 2. Support to set threshold according to StorageType and work. > 3. Add an option to prohibit nodes below the threshold from joining the > Source set. This is to allow nodes with high utilization to transfer data as > soon as possible, which is good for balance. > 4. Add new support. If there are a lot of datanode usage in the cluster, it > should remain unchanged. For example, the utilization rate of 40% of the > nodes in the cluster is 75% to 80%, and these nodes should not join the > Source set. Of course this support needs to be specified by the user at > runtime. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16614) Improve balancer operation strategy and performance
JiangHua Zhu created HDFS-16614: --- Summary: Improve balancer operation strategy and performance Key: HDFS-16614 URL: https://issues.apache.org/jira/browse/HDFS-16614 Project: Hadoop HDFS Issue Type: Improvement Components: balancer & mover, namenode Affects Versions: 3.3.0 Reporter: JiangHua Zhu Attachments: image-2022-06-02-13-18-33-213.png When the Balancer program is run, it does some work in the following order: 1. Obtain available datanode information from NameNode. 2. Classify and calculate the average utilization according to StorageType. Here, some sets will be obtained in combination with the set thresholds: overUtilized, aboveAvgUtilized, belowAvgUtilized, and underUtilized. 3. According to some calculations, the source and target related to the transfer data are obtained. The source is used for the source end, and the target is used for the data receiving end. 4. Start the data transfer work in parallel. In this process, run iteratively. In this process, the threshold is unified and applied to all StorageTypes, which seems to be a bit rough, because one of the StorageTypes cannot be distinguished, which is based on the currently supported heterogeneous storage. There is an online cluster with more than 2000 nodes, and there is an imbalance in node storage. E.g: !image-2022-06-02-13-18-33-213.png! Here, the average utilization of the cluster is 78%, but the utilization of most nodes is between 85% and 90%. When the balancer is turned on, we find that 85% of the nodes are working as sources. In this case, we think it is not reasonable, because it will occupy more network resources in the cluster, and it will be beneficial to the normal work of the cluster to do some effective restrictions. So here are some changes to make: 1. When the balancer is running, it should try to prompt the threshold related to StorageType. For example [[DISK, 10%], [SSD, 8%]...] 2. Support to set threshold according to StorageType and work. 3. Add an option to prohibit nodes below the threshold from joining the Source set. This is to allow nodes with high utilization to transfer data as soon as possible, which is good for balance. 4. Add new support. If there are a lot of datanode usage in the cluster, it should remain unchanged. For example, the utilization rate of 40% of the nodes in the cluster is 75% to 80%, and these nodes should not join the Source set. Of course this support needs to be specified by the user at runtime. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16614) Improve balancer operation strategy and performance
[ https://issues.apache.org/jira/browse/HDFS-16614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangHua Zhu reassigned HDFS-16614: --- Assignee: JiangHua Zhu > Improve balancer operation strategy and performance > --- > > Key: HDFS-16614 > URL: https://issues.apache.org/jira/browse/HDFS-16614 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover, namenode >Affects Versions: 3.3.0 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Attachments: image-2022-06-02-13-18-33-213.png > > > When the Balancer program is run, it does some work in the following order: > 1. Obtain available datanode information from NameNode. > 2. Classify and calculate the average utilization according to StorageType. > Here, some sets will be obtained in combination with the set thresholds: > overUtilized, aboveAvgUtilized, belowAvgUtilized, and underUtilized. > 3. According to some calculations, the source and target related to the > transfer data are obtained. The source is used for the source end, and the > target is used for the data receiving end. > 4. Start the data transfer work in parallel. > In this process, run iteratively. In this process, the threshold is unified > and applied to all StorageTypes, which seems to be a bit rough, because one > of the StorageTypes cannot be distinguished, which is based on the currently > supported heterogeneous storage. > There is an online cluster with more than 2000 nodes, and there is an > imbalance in node storage. E.g: > !image-2022-06-02-13-18-33-213.png! > Here, the average utilization of the cluster is 78%, but the utilization of > most nodes is between 85% and 90%. When the balancer is turned on, we find > that 85% of the nodes are working as sources. In this case, we think it is > not reasonable, because it will occupy more network resources in the cluster, > and it will be beneficial to the normal work of the cluster to do some > effective restrictions. > So here are some changes to make: > 1. When the balancer is running, it should try to prompt the threshold > related to StorageType. For example [[DISK, 10%], [SSD, 8%]...] > 2. Support to set threshold according to StorageType and work. > 3. Add an option to prohibit nodes below the threshold from joining the > Source set. This is to allow nodes with high utilization to transfer data as > soon as possible, which is good for balance. > 4. Add new support. If there are a lot of datanode usage in the cluster, it > should remain unchanged. For example, the utilization rate of 40% of the > nodes in the cluster is 75% to 80%, and these nodes should not join the > Source set. Of course this support needs to be specified by the user at > runtime. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16595) Slow peer metrics - add median, mad and upper latency limits
[
https://issues.apache.org/jira/browse/HDFS-16595?focusedWorklogId=777232&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-777232
]
ASF GitHub Bot logged work on HDFS-16595:
-
Author: ASF GitHub Bot
Created on: 02/Jun/22 01:20
Start Date: 02/Jun/22 01:20
Worklog Time Spent: 10m
Work Description: virajjasani commented on code in PR #4357:
URL: https://github.com/apache/hadoop/pull/4357#discussion_r887419980
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java:
##
@@ -853,11 +854,14 @@ public static List
convertSlowPeerInfo(
List slowPeerInfoProtos =
new ArrayList<>(slowPeers.getSlowPeers().size());
-for (Map.Entry entry :
-slowPeers.getSlowPeers().entrySet()) {
- slowPeerInfoProtos.add(SlowPeerReportProto.newBuilder()
+for (Map.Entry entry :
slowPeers.getSlowPeers().entrySet()) {
+ OutlierMetrics outlierMetrics = entry.getValue();
+ slowPeerInfoProtos.add(
+ SlowPeerReportProto.newBuilder()
.setDataNodeId(entry.getKey())
- .setAggregateLatency(entry.getValue())
+ .setAggregateLatency(outlierMetrics.getActualLatency())
+
.setMedian(outlierMetrics.getMedian()).setMad(outlierMetrics.getMad())
Review Comment:
Done, thanks for the suggestion.
Issue Time Tracking
---
Worklog Id: (was: 777232)
Time Spent: 2.5h (was: 2h 20m)
> Slow peer metrics - add median, mad and upper latency limits
>
>
> Key: HDFS-16595
> URL: https://issues.apache.org/jira/browse/HDFS-16595
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Slow datanode metrics include slow node and it's reporting node details. With
> HDFS-16582, we added the aggregate latency that is perceived by the reporting
> nodes.
> In order to get more insights into how the outlier slownode's latencies
> differ from the rest of the nodes, we should also expose median, median
> absolute deviation and the calculated upper latency limit details.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16595) Slow peer metrics - add median, mad and upper latency limits
[
https://issues.apache.org/jira/browse/HDFS-16595?focusedWorklogId=777224&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-777224
]
ASF GitHub Bot logged work on HDFS-16595:
-
Author: ASF GitHub Bot
Created on: 02/Jun/22 00:33
Start Date: 02/Jun/22 00:33
Worklog Time Spent: 10m
Work Description: jojochuang commented on code in PR #4357:
URL: https://github.com/apache/hadoop/pull/4357#discussion_r887401881
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java:
##
@@ -853,11 +854,14 @@ public static List
convertSlowPeerInfo(
List slowPeerInfoProtos =
new ArrayList<>(slowPeers.getSlowPeers().size());
-for (Map.Entry entry :
-slowPeers.getSlowPeers().entrySet()) {
- slowPeerInfoProtos.add(SlowPeerReportProto.newBuilder()
+for (Map.Entry entry :
slowPeers.getSlowPeers().entrySet()) {
+ OutlierMetrics outlierMetrics = entry.getValue();
+ slowPeerInfoProtos.add(
+ SlowPeerReportProto.newBuilder()
.setDataNodeId(entry.getKey())
- .setAggregateLatency(entry.getValue())
+ .setAggregateLatency(outlierMetrics.getActualLatency())
+
.setMedian(outlierMetrics.getMedian()).setMad(outlierMetrics.getMad())
Review Comment:
move .setMad to its own line?
```suggestion
.setMedian(outlierMetrics.getMedian())
.setMad(outlierMetrics.getMad())
```
Issue Time Tracking
---
Worklog Id: (was: 777224)
Time Spent: 2h 20m (was: 2h 10m)
> Slow peer metrics - add median, mad and upper latency limits
>
>
> Key: HDFS-16595
> URL: https://issues.apache.org/jira/browse/HDFS-16595
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Slow datanode metrics include slow node and it's reporting node details. With
> HDFS-16582, we added the aggregate latency that is perceived by the reporting
> nodes.
> In order to get more insights into how the outlier slownode's latencies
> differ from the rest of the nodes, we should also expose median, median
> absolute deviation and the calculated upper latency limit details.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16610) Make fsck read timeout configurable
[ https://issues.apache.org/jira/browse/HDFS-16610?focusedWorklogId=777124&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-777124 ] ASF GitHub Bot logged work on HDFS-16610: - Author: ASF GitHub Bot Created on: 01/Jun/22 19:36 Start Date: 01/Jun/22 19:36 Worklog Time Spent: 10m Work Description: sodonnel merged PR #4384: URL: https://github.com/apache/hadoop/pull/4384 Issue Time Tracking --- Worklog Id: (was: 777124) Time Spent: 1h 50m (was: 1h 40m) > Make fsck read timeout configurable > --- > > Key: HDFS-16610 > URL: https://issues.apache.org/jira/browse/HDFS-16610 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > In a cluster with a lot of small files, we encountered a case where fsck was > very slow. I believe it is due to contention with many other threads reading > / writing data on the cluster. > Sometimes fsck does not report any progress for more than 60 seconds and the > client times out. Currently the connect and read timeout are hardcoded to 60 > seconds. This change is to make them configurable. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16610) Make fsck read timeout configurable
[ https://issues.apache.org/jira/browse/HDFS-16610?focusedWorklogId=777123&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-777123 ] ASF GitHub Bot logged work on HDFS-16610: - Author: ASF GitHub Bot Created on: 01/Jun/22 19:35 Start Date: 01/Jun/22 19:35 Worklog Time Spent: 10m Work Description: sodonnel commented on PR #4384: URL: https://github.com/apache/hadoop/pull/4384#issuecomment-1144054446 Good build this time, so I will commit. Thanks all for the reviews. Issue Time Tracking --- Worklog Id: (was: 777123) Time Spent: 1h 40m (was: 1.5h) > Make fsck read timeout configurable > --- > > Key: HDFS-16610 > URL: https://issues.apache.org/jira/browse/HDFS-16610 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > In a cluster with a lot of small files, we encountered a case where fsck was > very slow. I believe it is due to contention with many other threads reading > / writing data on the cluster. > Sometimes fsck does not report any progress for more than 60 seconds and the > client times out. Currently the connect and read timeout are hardcoded to 60 > seconds. This change is to make them configurable. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16610) Make fsck read timeout configurable
[
https://issues.apache.org/jira/browse/HDFS-16610?focusedWorklogId=777018&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-777018
]
ASF GitHub Bot logged work on HDFS-16610:
-
Author: ASF GitHub Bot
Created on: 01/Jun/22 17:06
Start Date: 01/Jun/22 17:06
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4384:
URL: https://github.com/apache/hadoop/pull/4384#issuecomment-1143886604
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 9s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 16s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 52s | | trunk passed |
| +1 :green_heart: | compile | 7m 2s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 6m 22s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 12s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 28s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 35s | | the patch passed |
| +1 :green_heart: | compile | 7m 39s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 7m 39s | | the patch passed |
| +1 :green_heart: | compile | 7m 27s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 7m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 26s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 50s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 9s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 38s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 1s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 28s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 331m 58s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 0s | | The patch does not
generate ASF License warnings. |
| | | 490m 17s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4384/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4384 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux bd88f84c1b08 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0ecf65945d28b2c3243495f826057f390eb1fad6 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20
[jira] [Resolved] (HDFS-16612) impove import * In HDFS Project
[ https://issues.apache.org/jira/browse/HDFS-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fanshilun resolved HDFS-16612. -- Resolution: Not A Problem > impove import * In HDFS Project > --- > > Key: HDFS-16612 > URL: https://issues.apache.org/jira/browse/HDFS-16612 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16602) Use "defined" directive along with #if
[
https://issues.apache.org/jira/browse/HDFS-16602?focusedWorklogId=776795&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-776795
]
ASF GitHub Bot logged work on HDFS-16602:
-
Author: ASF GitHub Bot
Created on: 01/Jun/22 11:04
Start Date: 01/Jun/22 11:04
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4371:
URL: https://github.com/apache/hadoop/pull/4371#issuecomment-1143459438
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 24m 44s | | trunk passed |
| -1 :x: | compile | 0m 45s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt)
| hadoop-hdfs-native-client in trunk failed. |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 52m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 23s | | the patch passed |
| -1 :x: | compile | 0m 28s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt)
| hadoop-hdfs-native-client in the patch failed. |
| -1 :x: | cc | 0m 28s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt)
| hadoop-hdfs-native-client in the patch failed. |
| -1 :x: | golang | 0m 28s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt)
| hadoop-hdfs-native-client in the patch failed. |
| -1 :x: | javac | 0m 28s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt)
| hadoop-hdfs-native-client in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 0m 37s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt)
| hadoop-hdfs-native-client in the patch failed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 95m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4371 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 24833ef1266f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b472514254aeaae500853979626918c80c8387b4 |
| Default Java | Debian-11.0.15+10-post-Debian-1deb10u1 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/testReport/ |
| Max. process+thread count | 698 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/console |
| vers
[jira] [Work logged] (HDFS-16602) Use "defined" directive along with #if
[
https://issues.apache.org/jira/browse/HDFS-16602?focusedWorklogId=776737&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-776737
]
ASF GitHub Bot logged work on HDFS-16602:
-
Author: ASF GitHub Bot
Created on: 01/Jun/22 09:28
Start Date: 01/Jun/22 09:28
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4371:
URL: https://github.com/apache/hadoop/pull/4371#issuecomment-1143358246
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 22m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 23m 14s | | trunk passed |
| +1 :green_heart: | compile | 4m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 47m 55s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 3m 46s | | the patch passed |
| +1 :green_heart: | cc | 3m 46s | | the patch passed |
| +1 :green_heart: | golang | 3m 46s | | the patch passed |
| +1 :green_heart: | javac | 3m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 33m 12s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not
generate ASF License warnings. |
| | | 131m 46s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4371 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 0700fbc3045c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b472514254aeaae500853979626918c80c8387b4 |
| Default Java | Red Hat, Inc.-1.8.0_312-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/testReport/ |
| Max. process+thread count | 698 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/console |
| versions | git=2.27.0 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
Issue Time Tracking
---
Worklog Id: (was: 776737)
Time Spent: 40m (was: 0.5h)
> Use "defined" directive along with #if
> --
>
> Key: HDFS-16602
> URL: https://issues.apache.org/jira/browse/HDFS-16602
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: libhdfscpp, pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The #if preprocessor directive expects a boolean expression. Thus, we need to
> use the "defined" directive as well to check if the macro has been defined.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16610) Make fsck read timeout configurable
[ https://issues.apache.org/jira/browse/HDFS-16610?focusedWorklogId=776723&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-776723 ] ASF GitHub Bot logged work on HDFS-16610: - Author: ASF GitHub Bot Created on: 01/Jun/22 08:55 Start Date: 01/Jun/22 08:55 Worklog Time Spent: 10m Work Description: sodonnel commented on PR #4384: URL: https://github.com/apache/hadoop/pull/4384#issuecomment-1143322132 I rebased against the latest trunk and force pushed it. Lets see if we get a better build this time. Issue Time Tracking --- Worklog Id: (was: 776723) Time Spent: 1h 20m (was: 1h 10m) > Make fsck read timeout configurable > --- > > Key: HDFS-16610 > URL: https://issues.apache.org/jira/browse/HDFS-16610 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Reporter: Stephen O'Donnell >Assignee: Stephen O'Donnell >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > In a cluster with a lot of small files, we encountered a case where fsck was > very slow. I believe it is due to contention with many other threads reading > / writing data on the cluster. > Sometimes fsck does not report any progress for more than 60 seconds and the > client times out. Currently the connect and read timeout are hardcoded to 60 > seconds. This change is to make them configurable. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16463) Make dirent cross platform compatible
[
https://issues.apache.org/jira/browse/HDFS-16463?focusedWorklogId=776700&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-776700
]
ASF GitHub Bot logged work on HDFS-16463:
-
Author: ASF GitHub Bot
Created on: 01/Jun/22 07:41
Start Date: 01/Jun/22 07:41
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4370:
URL: https://github.com/apache/hadoop/pull/4370#issuecomment-1143228728
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 46m 8s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 55s | | trunk passed |
| +1 :green_heart: | compile | 3m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 67m 37s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | | the patch passed |
| +1 :green_heart: | compile | 3m 47s | | the patch passed |
| -1 :x: | cc | 3m 47s |
[/results-compile-cc-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4370/7/artifact/out/results-compile-cc-hadoop-hdfs-project_hadoop-hdfs-native-client.txt)
| hadoop-hdfs-project_hadoop-hdfs-native-client generated 2 new + 8 unchanged
- 0 fixed = 10 total (was 8) |
| +1 :green_heart: | golang | 3m 47s | | the patch passed |
| +1 :green_heart: | javac | 3m 47s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4370/7/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 5m 2s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4370/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt)
| hadoop-hdfs-native-client in the patch failed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 148m 51s | | |
| Reason | Tests |
|---:|:--|
| Failed CTEST tests | test_test_libhdfs_ops_hdfs_static |
| | test_test_libhdfs_threaded_hdfs_static |
| | test_test_libhdfs_zerocopy_hdfs_static |
| | test_test_native_mini_dfs |
| | test_libhdfs_threaded_hdfspp_test_shim_static |
| | test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static |
| | libhdfs_mini_stress_valgrind_hdfspp_test_static |
| | memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static |
| | test_libhdfs_mini_stress_hdfspp_test_shim_static |
| | test_hdfs_ext_hdfspp_test_shim_static |
| | x_platform_dirent_c_test |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4370/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4370 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 628acf824e94 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 27b3dc8c60092c35879010ad8f0a55eef265625a |
| Default Java | Red Hat, Inc.-1.8.0_332-b09 |
| CTEST |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4370/7/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4370/7/testReport/ |
| Max. process+thread count | 600 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project
[jira] [Work logged] (HDFS-16598) All datanodes [DatanodeInfoWithStorage[127.0.0.1:57448,DS-1b5f7e33-a2bf-4edc-9122-a74c995a99f5,DISK]] are bad. Aborting...
[
https://issues.apache.org/jira/browse/HDFS-16598?focusedWorklogId=776694&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-776694
]
ASF GitHub Bot logged work on HDFS-16598:
-
Author: ASF GitHub Bot
Created on: 01/Jun/22 07:20
Start Date: 01/Jun/22 07:20
Worklog Time Spent: 10m
Work Description: ZanderXu commented on PR #4366:
URL: https://github.com/apache/hadoop/pull/4366#issuecomment-1143208824
getReplicaInfo(ExtendedBlock b) will check gs, and getReplicaInfo(String
bpid, long blkid) will not check the gs.
Issue Time Tracking
---
Worklog Id: (was: 776694)
Time Spent: 1h 10m (was: 1h)
> All datanodes
> [DatanodeInfoWithStorage[127.0.0.1:57448,DS-1b5f7e33-a2bf-4edc-9122-a74c995a99f5,DISK]]
> are bad. Aborting...
> --
>
> Key: HDFS-16598
> URL: https://issues.apache.org/jira/browse/HDFS-16598
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> org.apache.hadoop.hdfs.testPipelineRecoveryOnRestartFailure failed with the
> stack like:
> {code:java}
> java.io.IOException: All datanodes
> [DatanodeInfoWithStorage[127.0.0.1:57448,DS-1b5f7e33-a2bf-4edc-9122-a74c995a99f5,DISK]]
> are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1667)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1601)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1587)
> at
> org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1371)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:674)
> {code}
> After tracing the root cause, this bug was introduced by
> [HDFS-16534|https://issues.apache.org/jira/browse/HDFS-16534]. Because the
> block GS of client may be smaller than DN when pipeline recovery failed.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16602) Use "defined" directive along with #if
[
https://issues.apache.org/jira/browse/HDFS-16602?focusedWorklogId=776693&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-776693
]
ASF GitHub Bot logged work on HDFS-16602:
-
Author: ASF GitHub Bot
Created on: 01/Jun/22 07:16
Start Date: 01/Jun/22 07:16
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4371:
URL: https://github.com/apache/hadoop/pull/4371#issuecomment-1143205258
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 42s | | trunk passed |
| +1 :green_heart: | compile | 4m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 61m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 3m 43s | | the patch passed |
| +1 :green_heart: | cc | 3m 43s | | the patch passed |
| +1 :green_heart: | golang | 3m 43s | | the patch passed |
| +1 :green_heart: | javac | 3m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 1s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 32m 47s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 122m 3s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4371 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 8f37b16f5a4a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b472514254aeaae500853979626918c80c8387b4 |
| Default Java | Red Hat, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/testReport/ |
| Max. process+thread count | 666 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4371/2/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
Issue Time Tracking
---
Worklog Id: (was: 776693)
Time Spent: 0.5h (was: 20m)
> Use "defined" directive along with #if
> --
>
> Key: HDFS-16602
> URL: https://issues.apache.org/jira/browse/HDFS-16602
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs++
>Affects Versions: 3.4.0
>Reporter: Gautham Banasandra
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: libhdfscpp, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The #if preprocessor directive expects a boolean expression. Thus, we need to
> use the "defined" directive as well to check if the macro has been defined.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16598) All datanodes [DatanodeInfoWithStorage[127.0.0.1:57448,DS-1b5f7e33-a2bf-4edc-9122-a74c995a99f5,DISK]] are bad. Aborting...
[
https://issues.apache.org/jira/browse/HDFS-16598?focusedWorklogId=776691&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-776691
]
ASF GitHub Bot logged work on HDFS-16598:
-
Author: ASF GitHub Bot
Created on: 01/Jun/22 07:14
Start Date: 01/Jun/22 07:14
Worklog Time Spent: 10m
Work Description: MingXiangLi commented on PR #4366:
URL: https://github.com/apache/hadoop/pull/4366#issuecomment-1143203294
Thanks for your feedback.But I don't quite understand why this PR can solves
the problem.It seems that you only change getReplicaInfo(x) to
getReplicaInfo(x,x), the logic in two method was almost same.
Issue Time Tracking
---
Worklog Id: (was: 776691)
Time Spent: 1h (was: 50m)
> All datanodes
> [DatanodeInfoWithStorage[127.0.0.1:57448,DS-1b5f7e33-a2bf-4edc-9122-a74c995a99f5,DISK]]
> are bad. Aborting...
> --
>
> Key: HDFS-16598
> URL: https://issues.apache.org/jira/browse/HDFS-16598
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> org.apache.hadoop.hdfs.testPipelineRecoveryOnRestartFailure failed with the
> stack like:
> {code:java}
> java.io.IOException: All datanodes
> [DatanodeInfoWithStorage[127.0.0.1:57448,DS-1b5f7e33-a2bf-4edc-9122-a74c995a99f5,DISK]]
> are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DataStreamer.handleBadDatanode(DataStreamer.java:1667)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1601)
> at
> org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1587)
> at
> org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1371)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:674)
> {code}
> After tracing the root cause, this bug was introduced by
> [HDFS-16534|https://issues.apache.org/jira/browse/HDFS-16534]. Because the
> block GS of client may be smaller than DN when pipeline recovery failed.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
