[
https://issues.apache.org/jira/browse/HDFS-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14240391#comment-14240391
]
Yongjun Zhang commented on HDFS-6833:
-------------------------------------
(sorry please ignore last one, submitted from wrong window)
HI [~sinchii],
Thanks for working on the new rev. It doesn't apply now due to the latest trunk
changes, but I looked at rev10 based on earlier commit and have the following
comments:
* {{deletingBlock = new ReplicaMap(this);}} is currently using {{this}}
(FsDataset object) as the mutex, we should replace the "this" with a different
one that's dedicated to synchronize access to {{deletingBlock}} to avoid
holding FsDataset object.
* There is no need to have {{private boolean scanning}} and in
{{DirectoryScanner}}. And we need to remove all references to it in the
changes. We don't care whether scanning is true or not, we just need to make
sure blocks deleted from disk are also removed from {{deleteingBlock}}
* There is no need for {{public boolean isDirectoryScannerInited()}} in
DataNode.
* Remove the local variable {{Set<Long> deletingBlockIds}} in method scan() ,
and we don't want to do the following because we do it from the other place
after deleting block files:
{code}
519 if (dataset.getNumDeletingBlocks(bpid) > 0) {
520 dataset.removeDeletedBlocks(bpid, deletingBlockIds, true);
521 }
{code}
* The following code that does the logging
{code}
statsRecord.deletingBlocks++;
LOG.info("Block file " + blockpoolReport[d].getBlockFile() + " is to be
deleted");
{code}
can be moved to the following place:
{code}
1570 ReplicaInfo removing = volumeMap.remove(bpid, invalidBlks[i]);
1571 if (datanode.isDirectoryScannerInited()) {
1572 deletingBlock.add(bpid, removing);
1573 }
{code}
And we should not check whether the directory scanner is running in the above
code because the directory scanner can start any time.
E.g., change the above 1570-1572 code to something like:
{code}
ReplicaInfo removing = volumeMap.remove(bpid, invalidBlks[i]);
deletingBlock.add(bpid, removing);
statsRecord.deletingBlocks++;
LOG.info("Block file " + blockpoolReport[d].getBlockFile() + " is to be
deleted"); ==> need to revise to only print at debug level?
{code}
* The following code need to be removed
{code}
} else {
LOG.info("Block file " + blockpoolReport[d].getBlockFile() + " is to be
deleted");
statsRecord.deletingBlocks++;
deletingBlockIds.add(info.getBlockId());
}
{code}
* The 1000 below is a bit high, I'd suggest we start with a bit smaller one,
such as 64. We might need to make it a configuration property later.
{code}
private static final int MAX_DELETED_BLOCKS = 1000;
{code}
BTW, are you still able to reproduce the issue and test the fix? If so, we can
test it out later after getting committer's review.
> DirectoryScanner should not register a deleting block with memory of DataNode
> -----------------------------------------------------------------------------
>
> Key: HDFS-6833
> URL: https://issues.apache.org/jira/browse/HDFS-6833
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 3.0.0, 2.5.0, 2.5.1
> Reporter: Shinichi Yamashita
> Assignee: Shinichi Yamashita
> Priority: Critical
> Attachments: HDFS-6833-10.patch, HDFS-6833-6-2.patch,
> HDFS-6833-6-3.patch, HDFS-6833-6.patch, HDFS-6833-7-2.patch,
> HDFS-6833-7.patch, HDFS-6833.8.patch, HDFS-6833.9.patch, HDFS-6833.patch,
> HDFS-6833.patch, HDFS-6833.patch, HDFS-6833.patch, HDFS-6833.patch
>
>
> When a block is deleted in DataNode, the following messages are usually
> output.
> {code}
> 2014-08-07 17:53:11,606 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Scheduling blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> for deletion
> 2014-08-07 17:53:11,617 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> {code}
> However, DirectoryScanner may be executed when DataNode deletes the block in
> the current implementation. And the following messsages are output.
> {code}
> 2014-08-07 17:53:30,519 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Scheduling blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> for deletion
> 2014-08-07 17:53:31,426 INFO
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
> BP-1887080305-172.28.0.101-1407398838872 Total blocks: 1, missing metadata
> files:0, missing block files:0, missing blocks in memory:1, mismatched
> blocks:0
> 2014-08-07 17:53:31,426 WARN
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added
> missing block to memory FinalizedReplica, blk_1073741825_1001, FINALIZED
> getNumBytes() = 21230663
> getBytesOnDisk() = 21230663
> getVisibleLength()= 21230663
> getVolume() = /hadoop/data1/dfs/data/current
> getBlockFile() =
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> unlinked =false
> 2014-08-07 17:53:31,531 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
> Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file
> /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
> {code}
> Deleting block information is registered in DataNode's memory.
> And when DataNode sends a block report, NameNode receives wrong block
> information.
> For example, when we execute recommission or change the number of
> replication, NameNode may delete the right block as "ExcessReplicate" by this
> problem.
> And "Under-Replicated Blocks" and "Missing Blocks" occur.
> When DataNode run DirectoryScanner, DataNode should not register a deleting
> block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
