[
https://issues.apache.org/jira/browse/HADOOP-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12657617#action_12657617
]
Hairong Kuang commented on HADOOP-3885:
---------------------------------------
The first deletion at 251_107_213 is recorded at ExcessReplicateMap. So it does
not count as a valid replica. NN still keeps it in blocksMap since NN has not
received confirmation from the datanode yet.
Were the following four deletions caused by the deletion of the file? When
deleting a file, NN deletes all replicas belonged to this file no matter if
they are already scheduled to delete or not. I think this is a safe solution
since NN has not heard any confirmation yet so NN is not sure if the replicas
are indeed deleted or not.
> Namenode blockMap not updated when datanode invalidates a block on heart beat
> -----------------------------------------------------------------------------
>
> Key: HADOOP-3885
> URL: https://issues.apache.org/jira/browse/HADOOP-3885
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Wei Xu
>
> Here are related logs. dn_10_251_107_213 deleted the block immediately after
> it received it (because of over-replication). However, the information is
> not reported to namenode and name node ask this DN to invalidate the block
> again, causing the error "Unexpected error trying to delete block
> blk_-1201258788540100829. BlockInfo not found in volumeMap.".
> dn_10_251_107_213 | 080722 145554 2497 INFO dfs.DataNode$DataXceiver:
> Receiving block blk_-1201258788540100829 src: /10.251.67.214:54770 dest:
> /10.251.67.214:50010
> dn_10_251_193_205 | 080722 145554 2468 INFO dfs.DataNode$DataXceiver:
> Receiving block blk_-1201258788540100829 src: /10.251.107.213:36718 dest:
> /10.251.107.213:50010
> dn_10_251_67_214 | 080722 145554 2306 INFO dfs.DataNode$DataXceiver:
> Receiving block blk_-1201258788540100829 src: /10.251.67.214:60599 dest:
> /10.251.67.214:50010
> nn_10_251_210_161 | 080722 145554 30 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.allocateBlock:
> /user/root/rand/_temporary/_task_200807221347_0001_m_000092_1/part-00092.
> blk_-1201258788540100829
> dn_10_251_107_213 | 080722 145656 2498 INFO dfs.DataNode$PacketResponder:
> Received block blk_-1201258788540100829 of size 29767698 from /10.251.67.214
> dn_10_251_193_205 | 080722 145656 2469 INFO dfs.DataNode$PacketResponder:
> Received block blk_-1201258788540100829 of size 29767698 from /10.251.107.213
> dn_10_251_67_214 | 080722 145656 2307 INFO dfs.DataNode$PacketResponder:
> Received block blk_-1201258788540100829 of size 29767698 from /10.251.67.214
> dn_10_251_107_213 | 080722 145656 2498 INFO dfs.DataNode$PacketResponder:
> PacketResponder 1 for block blk_-1201258788540100829 terminating
> dn_10_251_193_205 | 080722 145656 2469 INFO dfs.DataNode$PacketResponder:
> PacketResponder 0 for block blk_-1201258788540100829 terminating
> dn_10_251_67_214 | 080722 145656 2307 INFO dfs.DataNode$PacketResponder:
> PacketResponder 2 for block blk_-1201258788540100829 terminating
> nn_10_251_210_161 | 080722 145656 19 INFO dfs.FSNamesystem: BLOCK* ask
> 10.251.67.214:50010 to replicate blk_-1201258788540100829 to datanode(s)
> 10.251.107.49:50010
> nn_10_251_210_161 | 080722 145656 31 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.251.67.214:50010 is added to
> blk_-1201258788540100829 size 29767698
> nn_10_251_210_161 | 080722 145656 31 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.251.107.213:50010 is added to
> blk_-1201258788540100829 size 29767698
> nn_10_251_210_161 | 080722 145656 26 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.251.193.205:50010 is added to
> blk_-1201258788540100829 size 29767698
> dn_10_251_107_49 | 080722 145658 2484 INFO dfs.DataNode$DataXceiver:
> Receiving block blk_-1201258788540100829 src: /10.251.67.214:42237 dest:
> /10.251.67.214:50010
> dn_10_251_67_214 | 080722 145658 18 INFO dfs.DataNode: 10.251.67.214:50010
> Starting thread to transfer block blk_-1201258788540100829 to
> 10.251.107.49:50010
> dn_10_251_107_49 | 080722 145705 2484 INFO dfs.DataNode$DataXceiver: Received
> block blk_-1201258788540100829 src: /10.251.67.214:42237 dest:
> /10.251.67.214:50010 of size 29767698
> dn_10_251_67_214 | 080722 145705 2342 INFO dfs.DataNode$DataTransfer:
> 10.251.67.214:50010:Transmitted block blk_-1201258788540100829 to
> /10.251.107.49:50010
> nn_10_251_210_161 | 080722 145705 34 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.251.107.49:50010 is added to
> blk_-1201258788540100829 size 29767698
> dn_10_251_107_213 | 080722 145710 19 INFO dfs.FSDataset: Deleting block
> blk_-1201258788540100829 file
> /mnt/hadoop/dfs/data/current/subdir11/blk_-1201258788540100829
> nn_10_251_210_161 | 080722 150220 34 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.delete: blk_-1201258788540100829 is added to invalidSet of
> 10.251.67.214:50010
> nn_10_251_210_161 | 080722 150220 34 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.delete: blk_-1201258788540100829 is added to invalidSet of
> 10.251.107.213:50010
> nn_10_251_210_161 | 080722 150220 34 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.delete: blk_-1201258788540100829 is added to invalidSet of
> 10.251.193.205:50010
> nn_10_251_210_161 | 080722 150220 34 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.delete: blk_-1201258788540100829 is added to invalidSet of
> 10.251.107.49:50010
> dn_10_251_193_205 | 080722 150224 19 INFO dfs.FSDataset: Deleting block
> blk_-1201258788540100829 file
> /mnt/hadoop/dfs/data/current/subdir51/blk_-1201258788540100829
> dn_10_251_107_49 | 080722 150226 19 INFO dfs.FSDataset: Deleting block
> blk_-1201258788540100829 file
> /mnt/hadoop/dfs/data/current/subdir24/blk_-1201258788540100829
> dn_10_251_107_213 | 080722 150228 19 WARN dfs.FSDataset: Unexpected error
> trying to delete block blk_-1201258788540100829. BlockInfo not found in
> volumeMap.
> dn_10_251_67_214 | 080722 150253 18 INFO dfs.FSDataset: Deleting block
> blk_-1201258788540100829 file
> /mnt/hadoop/dfs/data/current/subdir43/blk_-1201258788540100829
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
