[
https://issues.apache.org/jira/browse/HADOOP-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12657622#action_12657622
]
Hairong Kuang commented on HADOOP-3885:
---------------------------------------
All I meant to say is that what's described in this jira is the expected
behavior of dfs. It does not cause any data lose except for some harmless
deletion warnings at datanodes. One improvement we could do is to make the
deletion confirmation back to NN sooner. Currently it waits for the next block
report to send the confirmation back. But in between a block deletion is
scheduled to remove and the deletion confirmation comes back, the reported
issue is likely to happen.
> Namenode blockMap not updated when datanode invalidates a block on heart beat
> -----------------------------------------------------------------------------
>
> Key: HADOOP-3885
> URL: https://issues.apache.org/jira/browse/HADOOP-3885
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Wei Xu
>
> Here are related logs. dn_10_251_107_213 deleted the block immediately after
> it received it (because of over-replication). However, the information is
> not reported to namenode and name node ask this DN to invalidate the block
> again, causing the error "Unexpected error trying to delete block
> blk_-1201258788540100829. BlockInfo not found in volumeMap.".
> dn_10_251_107_213 | 080722 145554 2497 INFO dfs.DataNode$DataXceiver:
> Receiving block blk_-1201258788540100829 src: /10.251.67.214:54770 dest:
> /10.251.67.214:50010
> dn_10_251_193_205 | 080722 145554 2468 INFO dfs.DataNode$DataXceiver:
> Receiving block blk_-1201258788540100829 src: /10.251.107.213:36718 dest:
> /10.251.107.213:50010
> dn_10_251_67_214 | 080722 145554 2306 INFO dfs.DataNode$DataXceiver:
> Receiving block blk_-1201258788540100829 src: /10.251.67.214:60599 dest:
> /10.251.67.214:50010
> nn_10_251_210_161 | 080722 145554 30 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.allocateBlock:
> /user/root/rand/_temporary/_task_200807221347_0001_m_000092_1/part-00092.
> blk_-1201258788540100829
> dn_10_251_107_213 | 080722 145656 2498 INFO dfs.DataNode$PacketResponder:
> Received block blk_-1201258788540100829 of size 29767698 from /10.251.67.214
> dn_10_251_193_205 | 080722 145656 2469 INFO dfs.DataNode$PacketResponder:
> Received block blk_-1201258788540100829 of size 29767698 from /10.251.107.213
> dn_10_251_67_214 | 080722 145656 2307 INFO dfs.DataNode$PacketResponder:
> Received block blk_-1201258788540100829 of size 29767698 from /10.251.67.214
> dn_10_251_107_213 | 080722 145656 2498 INFO dfs.DataNode$PacketResponder:
> PacketResponder 1 for block blk_-1201258788540100829 terminating
> dn_10_251_193_205 | 080722 145656 2469 INFO dfs.DataNode$PacketResponder:
> PacketResponder 0 for block blk_-1201258788540100829 terminating
> dn_10_251_67_214 | 080722 145656 2307 INFO dfs.DataNode$PacketResponder:
> PacketResponder 2 for block blk_-1201258788540100829 terminating
> nn_10_251_210_161 | 080722 145656 19 INFO dfs.FSNamesystem: BLOCK* ask
> 10.251.67.214:50010 to replicate blk_-1201258788540100829 to datanode(s)
> 10.251.107.49:50010
> nn_10_251_210_161 | 080722 145656 31 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.251.67.214:50010 is added to
> blk_-1201258788540100829 size 29767698
> nn_10_251_210_161 | 080722 145656 31 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.251.107.213:50010 is added to
> blk_-1201258788540100829 size 29767698
> nn_10_251_210_161 | 080722 145656 26 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.251.193.205:50010 is added to
> blk_-1201258788540100829 size 29767698
> dn_10_251_107_49 | 080722 145658 2484 INFO dfs.DataNode$DataXceiver:
> Receiving block blk_-1201258788540100829 src: /10.251.67.214:42237 dest:
> /10.251.67.214:50010
> dn_10_251_67_214 | 080722 145658 18 INFO dfs.DataNode: 10.251.67.214:50010
> Starting thread to transfer block blk_-1201258788540100829 to
> 10.251.107.49:50010
> dn_10_251_107_49 | 080722 145705 2484 INFO dfs.DataNode$DataXceiver: Received
> block blk_-1201258788540100829 src: /10.251.67.214:42237 dest:
> /10.251.67.214:50010 of size 29767698
> dn_10_251_67_214 | 080722 145705 2342 INFO dfs.DataNode$DataTransfer:
> 10.251.67.214:50010:Transmitted block blk_-1201258788540100829 to
> /10.251.107.49:50010
> nn_10_251_210_161 | 080722 145705 34 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.addStoredBlock: blockMap updated: 10.251.107.49:50010 is added to
> blk_-1201258788540100829 size 29767698
> dn_10_251_107_213 | 080722 145710 19 INFO dfs.FSDataset: Deleting block
> blk_-1201258788540100829 file
> /mnt/hadoop/dfs/data/current/subdir11/blk_-1201258788540100829
> nn_10_251_210_161 | 080722 150220 34 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.delete: blk_-1201258788540100829 is added to invalidSet of
> 10.251.67.214:50010
> nn_10_251_210_161 | 080722 150220 34 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.delete: blk_-1201258788540100829 is added to invalidSet of
> 10.251.107.213:50010
> nn_10_251_210_161 | 080722 150220 34 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.delete: blk_-1201258788540100829 is added to invalidSet of
> 10.251.193.205:50010
> nn_10_251_210_161 | 080722 150220 34 INFO dfs.FSNamesystem: BLOCK*
> NameSystem.delete: blk_-1201258788540100829 is added to invalidSet of
> 10.251.107.49:50010
> dn_10_251_193_205 | 080722 150224 19 INFO dfs.FSDataset: Deleting block
> blk_-1201258788540100829 file
> /mnt/hadoop/dfs/data/current/subdir51/blk_-1201258788540100829
> dn_10_251_107_49 | 080722 150226 19 INFO dfs.FSDataset: Deleting block
> blk_-1201258788540100829 file
> /mnt/hadoop/dfs/data/current/subdir24/blk_-1201258788540100829
> dn_10_251_107_213 | 080722 150228 19 WARN dfs.FSDataset: Unexpected error
> trying to delete block blk_-1201258788540100829. BlockInfo not found in
> volumeMap.
> dn_10_251_67_214 | 080722 150253 18 INFO dfs.FSDataset: Deleting block
> blk_-1201258788540100829 file
> /mnt/hadoop/dfs/data/current/subdir43/blk_-1201258788540100829
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
