[jira] [Commented] (HDFS-2251) Namenode does not recognize incorrectly sized blocks
[ https://issues.apache.org/jira/browse/HDFS-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14068749#comment-14068749 ] Allen Wittenauer commented on HDFS-2251: Despite Brian saying that we could close this out, it'd be good if we could verify that this is fixed. Namenode does not recognize incorrectly sized blocks Key: HDFS-2251 URL: https://issues.apache.org/jira/browse/HDFS-2251 Project: Hadoop HDFS Issue Type: Bug Reporter: Brian Bockelman We had a lot of file system corruption resulting in incorrectly sized blocks (on disk, they're truncated to 192KB when they should be 64MB). However, I cannot make Hadoop realize that these blocks are incorrectly sized. When I try to drain off the node, I get the following messages: 2008-10-29 18:46:51,293 WARN org.apache.hadoop.fs.FSNamesystem: Inconsistent size for block blk_-4403534125663454855_9937 reported from 172.16.1.150:50010 current size is 67108864 reported size is 196608 Here 172.16.1.150 is not the node which has the problematic block, but the destination of the file transfer. I propose that Hadoop should either: a) Upon startup, make sure that all blocks are properly sized (pro: rather cheap check; con: doesn't catch any truncations which happen while on disk) b) Upon detecting the incorrectly sized copy, Hadoop should ask the source of the block to perform a block verification. Thanks, Brian -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-2251) Namenode does not recognize incorrectly sized blocks
[ https://issues.apache.org/jira/browse/HDFS-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117575#comment-13117575 ] Brian Bockelman commented on HDFS-2251: --- Hi Uma, This is actually a super old ticket (Hadoop 0.18). I have no clue if it is still applicable to the existing code base (it might have gotten added in the last 3 years). However, it's a really easy way to detect a common corruption case - the advice still applies. Won't be offended if you just wanted to close it out until I see the issue again. Brian Namenode does not recognize incorrectly sized blocks Key: HDFS-2251 URL: https://issues.apache.org/jira/browse/HDFS-2251 Project: Hadoop HDFS Issue Type: Bug Reporter: Brian Bockelman We had a lot of file system corruption resulting in incorrectly sized blocks (on disk, they're truncated to 192KB when they should be 64MB). However, I cannot make Hadoop realize that these blocks are incorrectly sized. When I try to drain off the node, I get the following messages: 2008-10-29 18:46:51,293 WARN org.apache.hadoop.fs.FSNamesystem: Inconsistent size for block blk_-4403534125663454855_9937 reported from 172.16.1.150:50010 current size is 67108864 reported size is 196608 Here 172.16.1.150 is not the node which has the problematic block, but the destination of the file transfer. I propose that Hadoop should either: a) Upon startup, make sure that all blocks are properly sized (pro: rather cheap check; con: doesn't catch any truncations which happen while on disk) b) Upon detecting the incorrectly sized copy, Hadoop should ask the source of the block to perform a block verification. Thanks, Brian -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2251) Namenode does not recognize incorrectly sized blocks
[ https://issues.apache.org/jira/browse/HDFS-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100254#comment-13100254 ] Uma Maheswara Rao G commented on HDFS-2251: --- Hi Brian, In which version of Hadoop you faced this? Did you observe any recovery (or) write failures? Thanks Uma Namenode does not recognize incorrectly sized blocks Key: HDFS-2251 URL: https://issues.apache.org/jira/browse/HDFS-2251 Project: Hadoop HDFS Issue Type: Bug Reporter: Brian Bockelman We had a lot of file system corruption resulting in incorrectly sized blocks (on disk, they're truncated to 192KB when they should be 64MB). However, I cannot make Hadoop realize that these blocks are incorrectly sized. When I try to drain off the node, I get the following messages: 2008-10-29 18:46:51,293 WARN org.apache.hadoop.fs.FSNamesystem: Inconsistent size for block blk_-4403534125663454855_9937 reported from 172.16.1.150:50010 current size is 67108864 reported size is 196608 Here 172.16.1.150 is not the node which has the problematic block, but the destination of the file transfer. I propose that Hadoop should either: a) Upon startup, make sure that all blocks are properly sized (pro: rather cheap check; con: doesn't catch any truncations which happen while on disk) b) Upon detecting the incorrectly sized copy, Hadoop should ask the source of the block to perform a block verification. Thanks, Brian -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
