[
https://issues.apache.org/jira/browse/HDFS-16043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691607#comment-17691607
]
kinit edited comment on HDFS-16043 at 2/21/23 1:42 PM:
-------------------------------------------------------
When Observer NN tail edit log , if a large delete occurs, Observer NN has the
same problem as Active NN (holding a long write lock when deleting a large
directory), so this case should also be considered. We've already encountered
this problem recently.
do not know what problems will occur if modify it as follows:
{code:java}
public void removeBlocksAndUpdateSafemodeTotal(BlocksMapUpdateInfo
blocks,boolean deleteAsync) {
assert namesystem.hasWriteLock();
// In the case that we are a Standby tailing edits from the
// active while in safe-mode, we need to track the total number
// of blocks and safe blocks in the system.
boolean trackBlockCounts = bmSafeMode.isSafeModeTrackingBlocks();
int numRemovedComplete = 0, numRemovedSafe = 0;
for (BlockInfo b : blocks.getToDeleteList()) {
if (trackBlockCounts) {
if (b.isComplete()) {
numRemovedComplete++;
if (hasMinStorage(b, b.numNodes())) {
numRemovedSafe++;
}
}
}
if (!deleteAsync){
removeBlock(b);
}
}
// delete block async
if (deleteAsync) {
this.getMarkedDeleteQueue().add(blocks.getToDeleteList());
}
if (trackBlockCounts) {
LOG.debug("Adjusting safe-mode totals for deletion."
+ "decreasing safeBlocks by {}, totalBlocks by {}",
numRemovedSafe, numRemovedComplete);
bmSafeMode.adjustBlockTotals(-numRemovedSafe, -numRemovedComplete);
}
} {code}
was (Author: mofei):
When Observer NN tail edit log , if a large delete occurs, Observer NN has the
same problem as Active NN (holding a long write lock when deleting a large
directory), so this case should also be considered. We've already encountered
this problem recently.
do not know what problems will occur if modify it as follows:
!image-2023-02-21-21-39-39-873.png|width=858,height=569!
> Add markedDeleteBlockScrubberThread to delete blocks asynchronously
> -------------------------------------------------------------------
>
> Key: HDFS-16043
> URL: https://issues.apache.org/jira/browse/HDFS-16043
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, namanode
> Affects Versions: 3.4.0
> Reporter: Xiangyi Zhu
> Assignee: Xiangyi Zhu
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Attachments: 20210527-after.svg, 20210527-before.svg
>
> Time Spent: 12.5h
> Remaining Estimate: 0h
>
> Add markedDeleteBlockScrubberThread to delete blocks asynchronously.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
