[
https://issues.apache.org/jira/browse/HDFS-17218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17775613#comment-17775613
]
ASF GitHub Bot commented on HDFS-17218:
---------------------------------------
zhangshuyan0 commented on code in PR #6176:
URL: https://github.com/apache/hadoop/pull/6176#discussion_r1360327000
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:
##########
@@ -1819,6 +1820,19 @@ void removeBlocksAssociatedTo(final DatanodeStorageInfo
storageInfo) {
storageInfo, node);
}
+ /** Remove the blocks to the given DatanodeDescriptor from InvalidateBlocks.
*/
+ void removeBlocksFromInvalidateBlocks(final DatanodeDescriptor node) {
+ assert namesystem.hasWriteLock();
+ invalidateBlocks.remove(node);
+ }
+
+ /** Remove the blocks to the given DatanodeDescriptor from
excessRedundancyMap. */
+ LightWeightHashSet<BlockInfo> removeBlocksFromExcessRedundancyMap(
Review Comment:
removeBlocksFromExcessRedundancyMap -> removeNodeFromExcessRedundancyMap
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:
##########
@@ -1819,6 +1820,19 @@ void removeBlocksAssociatedTo(final DatanodeStorageInfo
storageInfo) {
storageInfo, node);
}
+ /** Remove the blocks to the given DatanodeDescriptor from InvalidateBlocks.
*/
+ void removeBlocksFromInvalidateBlocks(final DatanodeDescriptor node) {
Review Comment:
Suggest modifying the method name:
removeBlocksFromInvalidateBlocks -> removeNodeFromInvalidateBlocks
> NameNode should remove its excess blocks from the ExcessRedundancyMap When a
> DN registers
> -----------------------------------------------------------------------------------------
>
> Key: HDFS-17218
> URL: https://issues.apache.org/jira/browse/HDFS-17218
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namanode
> Reporter: Haiyang Hu
> Assignee: Haiyang Hu
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-10-12-15-52-52-336.png
>
>
> Currently found that DN will lose all pending DNA_INVALIDATE blocks if it
> restarts.
> *Root case*
> Current DN enables asynchronously deletion, it have many pending deletion
> blocks in memory.
> when DN restarts, these cached blocks may be lost. it causes some blocks in
> the excess map in the namenode to be leaked and this will result in many
> blocks having more replicas then expected.
> *solution*
> Consider NameNode should remove its excess blocks from the
> ExcessRedundancyMap When a DN registers,
> this approach will ensure that when processing the DN's full block report,
> the 'processExtraRedundancy' can be performed according to the actual of the
> blocks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
