tangshangwen created HDFS-9717: ---------------------------------- Summary: NameNode can not update the status of bad block Key: HDFS-9717 URL: https://issues.apache.org/jira/browse/HDFS-9717 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.2.0 Reporter: tangshangwen Assignee: tangshangwen
In our cluster,some users set the number of replicas of file to 1, then back to 2,the file cannot be read,but the NameNode think it is healthy {noformat} /user/username/dt=2015-11-30/dp=16/part-r-00063.lzo 1513716944 bytes, 12 block(s): Under replicated BP-1422437282658:blk_1897961957_824575827. Target Replicas is 2 but found 1 replica(s). Replica placement policy is violated for BP-1422437282658:blk_1897961957_824575827. Block should be additionally replicated on 1 more rack (s). 0. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961824_824575694 len=134217728 repl=2 [host1:50010, host2:50010] 1. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961957_824575827 len=134217728 repl=1 [host3:50010] 2. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897962047_824575917 len=134217728 repl=2 [host4:50010, host1:50010] ...... Status: HEALTHY Total size: 1513716944 B Total dirs: 0 Total files: 1 Total symlinks: 0 Total blocks (validated): 12 (avg. block size 126143078 B) Minimally replicated blocks: 12 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 1 (8.333333 %) Mis-replicated blocks: 1 (8.333333 %) Default replication factor: 3 Average block replication: 1.9166666 Corrupt blocks: 0 Missing replicas: 1 (4.1666665 %) Number of data-nodes: xxxx Number of racks: xxx FSCK ended at Thu Jan 28 10:27:49 CST 2016 in 0 milliseconds {noformat} But the replica on the datanode has been damaged, can't read,this is datanode log {noformat} 2016-01-23 06:34:42,737 WARN org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: First Verification failed for BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961957_824575827 java.io.IOException: Input/output error at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:272) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:529) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:710) at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyBlock(BlockPoolSliceScanner.java:427) at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyFirstBlock(BlockPoolSliceScanner.java:506) at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scan(BlockPoolSliceScanner.java:667) at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scanBlockPoolSlice(BlockPoolSliceScanner.java:633) at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:101) at java.lang.Thread.run(Thread.java:745) ------------------------------ 2016-01-28 10:28:37,874 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(host1, storageID=DS-1450783279-xxx.xxx.xxx.xxx-50010-1432889625435 , infoPort=50075, ipcPort=50020, storageInfo=lv=-47;cid=CID-3f36397d-b160-4414-b7e4-f37b72e96d53;nsid=1992344832;c=0):Failed to transfer BP-1337805335-xxx.xxx.xxx.xxx-142243 7282658:blk_1897961957_824575827 to xxx.xxx.xxx.xxx:50010 got java.net.SocketException: Original Exception : java.io.IOException: Input/output error at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:546) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:710) at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1520) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Input/output error {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)