[ https://issues.apache.org/jira/browse/HDFS-9717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Li updated HDFS-9717: ----------------------------- Component/s: namenode > NameNode can not update the status of bad block > ----------------------------------------------- > > Key: HDFS-9717 > URL: https://issues.apache.org/jira/browse/HDFS-9717 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode > Affects Versions: 2.2.0 > Reporter: tangshangwen > Assignee: tangshangwen > > In our cluster,some users set the number of replicas of file to 1, then back > to 2,the file cannot be read,but the NameNode think it is healthy > {noformat} > /user/username/dt=2015-11-30/dp=16/part-r-00063.lzo 1513716944 bytes, 12 > block(s): Under replicated BP-1422437282658:blk_1897961957_824575827. Target > Replicas is 2 but found 1 replica(s). > Replica placement policy is violated for > BP-1422437282658:blk_1897961957_824575827. Block should be additionally > replicated on 1 more rack > (s). > 0. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961824_824575694 > len=134217728 repl=2 [host1:50010, host2:50010] > 1. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961957_824575827 > len=134217728 repl=1 [host3:50010] > 2. BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897962047_824575917 > len=134217728 repl=2 [host4:50010, host1:50010] > ...... > Status: HEALTHY > Total size: 1513716944 B > Total dirs: 0 > Total files: 1 > Total symlinks: 0 > Total blocks (validated): 12 (avg. block size 126143078 B) > Minimally replicated blocks: 12 (100.0 %) > Over-replicated blocks: 0 (0.0 %) > Under-replicated blocks: 1 (8.333333 %) > Mis-replicated blocks: 1 (8.333333 %) > Default replication factor: 3 > Average block replication: 1.9166666 > Corrupt blocks: 0 > Missing replicas: 1 (4.1666665 %) > Number of data-nodes: xxxx > Number of racks: xxx > FSCK ended at Thu Jan 28 10:27:49 CST 2016 in 0 milliseconds > {noformat} > But the replica on the datanode has been damaged, can't read,this is > datanode log > {noformat} > 2016-01-23 06:34:42,737 WARN > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: First > Verification failed for > BP-1337805335-xxx.xxx.xxx.xxx-1422437282658:blk_1897961957_824575827 > > > java.io.IOException: Input/output error > > > at java.io.FileInputStream.readBytes(Native Method) > > > at java.io.FileInputStream.read(FileInputStream.java:272) > > > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192) > > > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:529) > > > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:710) > > > at > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyBlock(BlockPoolSliceScanner.java:427) > > at > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyFirstBlock(BlockPoolSliceScanner.java:506) > > at > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scan(BlockPoolSliceScanner.java:667) > > at > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scanBlockPoolSlice(BlockPoolSliceScanner.java:633) > > at > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:101) > > at java.lang.Thread.run(Thread.java:745) > ------------------------------ > 2016-01-28 10:28:37,874 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: > DatanodeRegistration(host1, > storageID=DS-1450783279-xxx.xxx.xxx.xxx-50010-1432889625435 > , infoPort=50075, ipcPort=50020, > storageInfo=lv=-47;cid=CID-3f36397d-b160-4414-b7e4-f37b72e96d53;nsid=1992344832;c=0):Failed > to transfer BP-1337805335-xxx.xxx.xxx.xxx-142243 > 7282658:blk_1897961957_824575827 to xxx.xxx.xxx.xxx:50010 got > java.net.SocketException: Original Exception : java.io.IOException: > Input/output error > at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) > at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) > at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) > at > org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:546) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:710) > at > org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1520) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: Input/output error > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)