(hadoop) branch HDFS-17384 updated: HDFS-17422. Enhance the stability of the unit test TestDFSAdmin (#6621). Contributed by lei w and Hualong Zhang. (#6657)
This is an automated email from the ASF dual-hosted git repository.
slfan1989 pushed a commit to branch HDFS-17384
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/HDFS-17384 by this push:
new 8a3eeca112f9 HDFS-17422. Enhance the stability of the unit test
TestDFSAdmin (#6621). Contributed by lei w and Hualong Zhang. (#6657)
8a3eeca112f9 is described below
commit 8a3eeca112f9286c0dad40de07ae54ee0b632c95
Author: ZanderXu
AuthorDate: Sat Mar 23 07:32:21 2024 +0800
HDFS-17422. Enhance the stability of the unit test TestDFSAdmin (#6621).
Contributed by lei w and Hualong Zhang. (#6657)
Reviewed-by: Dinesh Chitlangia
Signed-off-by: Shilun Fan
---
.../test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java | 10 --
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java
index 451a419aa140..392866e86ea2 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java
@@ -1353,8 +1353,14 @@ public class TestDFSAdmin {
decommissioningNode1.getIpcPort();
String node2Addr = decommissioningNode2.getIpAddr() + ":" +
decommissioningNode2.getIpcPort();
- assertTrue(outsForFinishReconf.get(0).contains(node1Addr)
- && outsForFinishReconf.get(0).contains(node2Addr));
+ int finishedReconfCount = 0;
+ for (String outMessage : outsForFinishReconf) {
+finishedReconfCount = outMessage.contains(node1Addr) ?
+finishedReconfCount + 1 : finishedReconfCount + 0;
+finishedReconfCount = outMessage.contains(node2Addr) ?
+finishedReconfCount + 1 : finishedReconfCount + 0;
+ }
+ assertTrue(finishedReconfCount == 2);
}
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop) branch trunk updated (8cd4704e0a1a -> a60b5e2de307)
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git from 8cd4704e0a1a HDFS-17430. RecoveringBlock will skip no live replicas when get block recovery command. (#6635) add a60b5e2de307 MAPREDUCE-7469. NNBench createControlFiles should use thread pool to improve performance. (#6463) Contributed by liuguanghua. No new revisions were added by this update. Summary of changes: .../test/java/org/apache/hadoop/hdfs/NNBench.java | 45 +- .../java/org/apache/hadoop/hdfs/TestNNBench.java | 20 ++ 2 files changed, 63 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-site) branch asf-site updated: Add Menu List 3.3.6. (#56) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hadoop-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 0f91b026b6 Add Menu List 3.3.6. (#56) Contributed by Shilun Fan. 0f91b026b6 is described below commit 0f91b026b6016a8aab95a7584214173f71d4b5cb Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Sat Mar 23 05:42:11 2024 +0800 Add Menu List 3.3.6. (#56) Contributed by Shilun Fan. Reviewed-by: Dinesh Chitlangia Signed-off-by: Shilun Fan --- content/bylaws.html| 2 ++ content/categories.html| 2 ++ content/committer_criteria.html| 2 ++ content/cve_list.html | 2 ++ content/description.html | 2 ++ content/index.html | 2 ++ content/issue_tracking.html| 2 ++ content/mailing_lists.html | 2 ++ content/modules.html | 2 ++ content/news.html | 2 ++ content/news/2008-07-xx-terabyte-sort.html | 2 ++ content/news/2008-11-xx-apachecon.html | 2 ++ content/news/2009-03-xx-apachecon-eu.html | 2 ++ content/news/2009-07-xx-subprojects.html | 2 ++ content/news/2010-05-xx-avro-hbase.html| 2 ++ content/news/2010-10-xx-hive-pig.html | 2 ++ content/news/2011-01-xx-zookeeper.html | 2 ++ content/news/2011-03-xx-award.html | 2 ++ content/news/2018-10-01-ozone-0.2.1-alpha.html | 2 ++ content/news/2018-11-22-ozone-0.3.0-alpha.html | 2 ++ content/news/2019-05-07-ozone-0.4.0-alpha.html | 2 ++ content/news/2019-10-13-ozone-0.4.1-alpha.html | 2 ++ content/news/2020-03-24-ozone-0.5.0-beta.html | 2 ++ content/news/2020-09-02-ozone-1.0.0.html | 2 ++ content/news/2021-04-17-ozone-1.1.0.html | 2 ++ content/news/2021-12-17-log4shell.html | 2 ++ content/news/page/2.html | 2 ++ content/privacy_policy.html| 2 ++ content/related.html | 2 ++ content/release.html | 2 ++ content/release/0.14.1.html| 2 ++ content/release/0.14.3.html| 2 ++ content/release/0.14.4.html| 2 ++ content/release/0.15.0.html| 2 ++ content/release/0.15.1.html| 2 ++ content/release/0.15.2.html| 2 ++ content/release/0.15.3.html| 2 ++ content/release/0.16.0.html| 2 ++ content/release/0.16.1.html| 2 ++ content/release/0.16.2.html| 2 ++ content/release/0.16.3.html| 2 ++ content/release/0.16.4.html| 2 ++ content/release/0.17.0.html| 2 ++ content/release/0.17.1.html| 2 ++ content/release/0.17.2.html| 2 ++ content/release/0.18.0.html| 2 ++ content/release/0.18.1.html| 2 ++ content/release/0.18.2.html| 2 ++ content/release/0.18.3.html| 2 ++ content/release/0.19.0.html| 2 ++ content/release/0.19.1.html| 2 ++ content/release/0.19.2.html| 2 ++ content/release/0.20.0.html| 2 ++ content/release/0.20.1.html| 2 ++ content/release/0.20.2.html| 2 ++ content/release/0.20.203.0.html| 2 ++ content/release/0.20.204.0.html| 2 ++ content/release/0.20.205.0.html| 2 ++ content/release/0.21.0.html| 2 ++ content/release/0.22.0.html| 2 ++ content/release/0.23.0.html| 2 ++ content/release/0.23.1.html| 2 ++ content/release/0.23.10.html | 2 ++ content/release/0.23.11.html | 2 ++ content/release/0.23.3.html| 2 ++ content/release/0.23.4.html| 2 ++ content/release/0.23.5.html| 2 ++ content/release/0.23.6.html| 2 ++ content/release/0.23.7.html| 2 ++ content/release/0.23.8.html| 2 ++ content/release/0.23.9.html| 2 ++ content/release/1.0.0.html | 2 ++ content/release/1.0.1.html | 2 ++ content/release/1.0.2.html | 2 ++ content/release/1.0.3.html | 2 ++ content/release/1.0.4.html | 2 ++ content/release/1.1.0.html | 2 ++ content/release/1.1.1.html
(hadoop) branch trunk updated: HDFS-17430. RecoveringBlock will skip no live replicas when get block recovery command. (#6635)
This is an automated email from the ASF dual-hosted git repository.
dineshc pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 8cd4704e0a1a HDFS-17430. RecoveringBlock will skip no live replicas
when get block recovery command. (#6635)
8cd4704e0a1a is described below
commit 8cd4704e0a1ab2ba941e296b8799e288f901bc4f
Author: huhaiyang
AuthorDate: Fri Mar 22 21:43:12 2024 +0800
HDFS-17430. RecoveringBlock will skip no live replicas when get block
recovery command. (#6635)
---
.../server/blockmanagement/DatanodeManager.java| 30 +++---
1 file changed, 21 insertions(+), 9 deletions(-)
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
index ebd2fa992e97..29327a1611c2 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
@@ -1729,12 +1729,13 @@ public class DatanodeManager {
" where it is not under construction.");
}
final DatanodeStorageInfo[] storages = uc.getExpectedStorageLocations();
- // Skip stale nodes during recovery
- final List recoveryLocations =
+ // Skip stale and dead nodes during recovery.
+ List recoveryLocations =
new ArrayList<>(storages.length);
- final List storageIdx = new ArrayList<>(storages.length);
+ List storageIdx = new ArrayList<>(storages.length);
for (int i = 0; i < storages.length; ++i) {
-if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
+if (!storages[i].getDatanodeDescriptor().isStale(staleInterval) &&
+storages[i].getDatanodeDescriptor().isAlive()) {
recoveryLocations.add(storages[i]);
storageIdx.add(i);
}
@@ -1747,20 +1748,31 @@ public class DatanodeManager {
ExtendedBlock primaryBlock = (copyOnTruncateRecovery) ?
new ExtendedBlock(blockPoolId, uc.getTruncateBlock()) :
new ExtendedBlock(blockPoolId, b);
- // If we only get 1 replica after eliminating stale nodes, choose all
+ // If we only get 1 replica after eliminating stale and dead nodes,
choose all live
// replicas for recovery and let the primary data node handle failures.
DatanodeInfo[] recoveryInfos;
if (recoveryLocations.size() > 1) {
if (recoveryLocations.size() != storages.length) {
- LOG.info("Skipped stale nodes for recovery : "
+ LOG.info("Skipped stale and dead nodes for recovery : "
+ (storages.length - recoveryLocations.size()));
}
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
} else {
-// If too many replicas are stale, then choose all replicas to
+// If too many replicas are stale, then choose all live replicas to
// participate in block recovery.
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
+recoveryLocations.clear();
+storageIdx.clear();
+for (int i = 0; i < storages.length; ++i) {
+ if (storages[i].getDatanodeDescriptor().isAlive()) {
+recoveryLocations.add(storages[i]);
+storageIdx.add(i);
+ }
+}
+if (recoveryLocations.size() != storages.length) {
+ LOG.info("Skipped dead nodes for recovery : {}",
+ storages.length - recoveryLocations.size());
+}
}
+ recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
RecoveringBlock rBlock;
if (truncateRecovery) {
Block recoveryBlock = (copyOnTruncateRecovery) ? b :
uc.getTruncateBlock();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
(hadoop-site) branch asf-site updated: Change Stable to hadoop 3.3.6. (#55) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hadoop-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 25658da254 Change Stable to hadoop 3.3.6. (#55) Contributed by Shilun Fan. 25658da254 is described below commit 25658da2544700b00aa2c1bb45c5299761fa62ee Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Fri Mar 22 19:53:40 2024 +0800 Change Stable to hadoop 3.3.6. (#55) Contributed by Shilun Fan. Reviewed-by: He Xiaoqiao Signed-off-by: Shilun Fan --- content/docs/stable3 | 2 +- src/release/3.3.6.md | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/content/docs/stable3 b/content/docs/stable3 index 902332bd22..b7086a22f6 12 --- a/content/docs/stable3 +++ b/content/docs/stable3 @@ -1 +1 @@ -r3.4.0 \ No newline at end of file +r3.3.6 \ No newline at end of file diff --git a/src/release/3.3.6.md b/src/release/3.3.6.md index 346e2bffbd..76756cb82e 100644 --- a/src/release/3.3.6.md +++ b/src/release/3.3.6.md @@ -1,6 +1,7 @@ --- title: Release 3.3.6 available date: 2023-06-23 +linked: true ---
(hadoop-site) branch asf-site updated: Update Doap Release 3.4.0. (#54) Contributed by Shilun Fan.
This is an automated email from the ASF dual-hosted git repository. slfan1989 pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/hadoop-site.git The following commit(s) were added to refs/heads/asf-site by this push: new 9cf727338b Update Doap Release 3.4.0. (#54) Contributed by Shilun Fan. 9cf727338b is described below commit 9cf727338b065af782f4edf7b877f60607268bee Author: slfan1989 <55643692+slfan1...@users.noreply.github.com> AuthorDate: Fri Mar 22 19:51:58 2024 +0800 Update Doap Release 3.4.0. (#54) Contributed by Shilun Fan. Reviewed-by: He Xiaoqiao Signed-off-by: Shilun Fan --- content/doap_Hadoop.rdf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/doap_Hadoop.rdf b/content/doap_Hadoop.rdf index 2b25fa687a..96e6644f49 100644 --- a/content/doap_Hadoop.rdf +++ b/content/doap_Hadoop.rdf @@ -37,8 +37,8 @@ Stable Release -2023-06-18 -3.3.6 +2024-03-17 +3.4.0 - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
