[ https://issues.apache.org/jira/browse/HDFS-16043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691607#comment-17691607 ]
kinit edited comment on HDFS-16043 at 2/21/23 1:42 PM: ------------------------------------------------------- When Observer NN tail edit log , if a large delete occurs, Observer NN has the same problem as Active NN (holding a long write lock when deleting a large directory), so this case should also be considered. We've already encountered this problem recently. do not know what problems will occur if modify it as follows: {code:java} public void removeBlocksAndUpdateSafemodeTotal(BlocksMapUpdateInfo blocks,boolean deleteAsync) { assert namesystem.hasWriteLock(); // In the case that we are a Standby tailing edits from the // active while in safe-mode, we need to track the total number // of blocks and safe blocks in the system. boolean trackBlockCounts = bmSafeMode.isSafeModeTrackingBlocks(); int numRemovedComplete = 0, numRemovedSafe = 0; for (BlockInfo b : blocks.getToDeleteList()) { if (trackBlockCounts) { if (b.isComplete()) { numRemovedComplete++; if (hasMinStorage(b, b.numNodes())) { numRemovedSafe++; } } } if (!deleteAsync){ removeBlock(b); } } // delete block async if (deleteAsync) { this.getMarkedDeleteQueue().add(blocks.getToDeleteList()); } if (trackBlockCounts) { LOG.debug("Adjusting safe-mode totals for deletion." + "decreasing safeBlocks by {}, totalBlocks by {}", numRemovedSafe, numRemovedComplete); bmSafeMode.adjustBlockTotals(-numRemovedSafe, -numRemovedComplete); } } {code} was (Author: mofei): When Observer NN tail edit log , if a large delete occurs, Observer NN has the same problem as Active NN (holding a long write lock when deleting a large directory), so this case should also be considered. We've already encountered this problem recently. do not know what problems will occur if modify it as follows: !image-2023-02-21-21-39-39-873.png|width=858,height=569! > Add markedDeleteBlockScrubberThread to delete blocks asynchronously > ------------------------------------------------------------------- > > Key: HDFS-16043 > URL: https://issues.apache.org/jira/browse/HDFS-16043 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namanode > Affects Versions: 3.4.0 > Reporter: Xiangyi Zhu > Assignee: Xiangyi Zhu > Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Attachments: 20210527-after.svg, 20210527-before.svg > > Time Spent: 12.5h > Remaining Estimate: 0h > > Add markedDeleteBlockScrubberThread to delete blocks asynchronously. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org