[jira] [Commented] (HDFS-17556) Avoid adding block to neededReconstruction repeatedly in decommission

[ASF GitHub Bot (Jira)]

    [ 
https://issues.apache.org/jira/browse/HDFS-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876518#comment-17876518
 ] 

ASF GitHub Bot commented on HDFS-17556:
---------------------------------------

haiyang1987 commented on code in PR #6896:
URL: https://github.com/apache/hadoop/pull/6896#discussion_r1730368502


##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java:
##########
@@ -2329,4 +2331,81 @@ public void delayDeleteReplica() {
       DataNodeFaultInjector.set(oldInjector);
     }
   }
+
+  private void waitForDecommissionedNodes(final DatanodeAdminManager 
dnAdminMgr,
+      final int trackedNumber)
+      throws TimeoutException, InterruptedException {
+    GenericTestUtils
+        .waitFor(() -> dnAdminMgr.getNumTrackedNodes() == trackedNumber,
+            100, 2000);
+  }
+
+  @Test(timeout = 360000)
+  public void testDecommissionWithPendingTimeout() throws Exception {
+    Configuration conf = new HdfsConfiguration();
+    conf.setBoolean(DFSConfigKeys.DFS_NAMENODE_REDUNDANCY_CONSIDERLOAD_KEY, 
false);
+
+    HostsFileWriter hostsFileWriter = new HostsFileWriter();
+    hostsFileWriter.initialize(conf, "work-dir/decommission");
+    conf.setInt(DFSConfigKeys.DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_KEY, 
1000);
+    conf.setInt(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, 1);
+    
conf.setInt(DFSConfigKeys.DFS_NAMENODE_RECONSTRUCTION_PENDING_TIMEOUT_SEC_KEY, 
2);
+    conf.setInt(DFSConfigKeys.DFS_NAMENODE_REDUNDANCY_INTERVAL_SECONDS_KEY, 
10);
+    conf.setInt(DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_INTERVAL_KEY, 1);
+    conf.setLong(DFSConfigKeys.DFS_DATANODE_BALANCE_BANDWIDTHPERSEC_KEY, 1);
+
+    MiniDFSCluster cluster = new 
MiniDFSCluster.Builder(conf).numDataNodes(2).build();

Review Comment:
   From [2357-2380], maybe can update logic
   ```
   
       MiniDFSCluster cluster = new 
MiniDFSCluster.Builder(conf).numDataNodes(2).build();
       try {
         cluster.waitActive();
         BlockManager blockManager = cluster.getNamesystem().getBlockManager();
         DatanodeManager dm = blockManager.getDatanodeManager();
         dm.setHeartbeatExpireInterval(3000);
         DistributedFileSystem fs = cluster.getFileSystem();
   
         // create a file
         Path filePath = new Path("/tmp.txt");
         DFSTestUtil.createFile(fs, filePath, 1024, (short) 1, 0L);
   
         LocatedBlocks blocks =
             NameNodeAdapter.getBlockLocations(cluster.getNameNode(), 
filePath.toString(), 0, 1);
         LocatedBlock locatedBlock = blocks.get(0);
         BlockInfo bi = 
blockManager.getStoredBlock(locatedBlock.getBlock().getLocalBlock());
         String dnName = locatedBlock.getLocations()[0].getXferAddr();
   
         ArrayList<String> dns = new ArrayList<String>(1);
   ```
   





> Avoid adding block to neededReconstruction repeatedly in decommission
> ---------------------------------------------------------------------
>
>                 Key: HDFS-17556
>                 URL: https://issues.apache.org/jira/browse/HDFS-17556
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namanode
>    Affects Versions: 3.5.0
>            Reporter: caozhiqiang
>            Assignee: caozhiqiang
>            Priority: Major
>              Labels: pull-request-available
>
> In decommission and maintenance process, before added to 
> BlockManager::neededReconstruction block will be check if it has been added. 
> The check contains if block is in BlockManager::neededReconstruction or in 
> PendingReconstructionBlocks::pendingReconstructions as below code. 
> But it also need to check if it is in 
> PendingReconstructionBlocks::timedOutItems. Or else 
> DatanodeAdminDefaultMonitor will add block to 
> BlockManager::neededReconstruction repeatedly if block time out in 
> PendingReconstructionBlocks::pendingReconstructions.
>  
> {code:java}
> if (!blockManager.neededReconstruction.contains(block) &&
>     blockManager.pendingReconstruction.getNumReplicas(block) == 0 &&
>     blockManager.isPopulatingReplQueues()) {
>   // Process these blocks only when active NN is out of safe mode.
>   blockManager.neededReconstruction.add(block,
>       liveReplicas, num.readOnlyReplicas(),
>       num.outOfServiceReplicas(),
>       blockManager.getExpectedRedundancyNum(block));
> } {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org