[ https://issues.apache.org/jira/browse/HDFS-7809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14326067#comment-14326067 ]
Kihwal Lee commented on HDFS-7809: ---------------------------------- Stack trace: {panel} 2015-02-13 01:07:45,628 \[org.apache.hadoop.hdfs.server.datanode.DataNode$2@278a83a0\] WARN datanode.DataNode: recoverBlocks FAILED: RecoveringBlock\{BP-xxxxx:blk_12345_10000; getBlockSize()=4150; corrupt=false; offset=-1; locs=\[1.2.3.4:1004, 1.2.3.5:1004, 1.2.3.6:1004\]\} org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.NSQuotaExceededException): Failed to record modification for snapshot: The NameSpace quota (directories and files) is exceeded: quota=50000 file count=50001 at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyNamespaceQuota(DirectoryWithQuotaFeature.java:138) at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:153) at org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.addSpaceConsumed(DirectoryWithQuotaFeature.java:96) at org.apache.hadoop.hdfs.server.namenode.INodeDirectory.addSpaceConsumed(INodeDirectory.java:136) at org.apache.hadoop.hdfs.server.namenode.INode.addSpaceConsumed2Parent(INode.java:484) at org.apache.hadoop.hdfs.server.namenode.INodeDirectory.addSpaceConsumed(INodeDirectory.java:138) at org.apache.hadoop.hdfs.server.namenode.INode.addSpaceConsumed2Parent(INode.java:484) at org.apache.hadoop.hdfs.server.namenode.INodeDirectory.addSpaceConsumed(INodeDirectory.java:138) at org.apache.hadoop.hdfs.server.namenode.INode.addSpaceConsumed2Parent(INode.java:484) at org.apache.hadoop.hdfs.server.namenode.INode.addSpaceConsumed(INode.java:474) at org.apache.hadoop.hdfs.server.namenode.snapshot.AbstractINodeDiffList.addDiff(AbstractINodeDiffList.java:125) at org.apache.hadoop.hdfs.server.namenode.snapshot.AbstractINodeDiffList.checkAndAddLatestSnapshotDiff(AbstractINodeDiffList.java:284) at org.apache.hadoop.hdfs.server.namenode.snapshot.AbstractINodeDiffList.saveSelf2Snapshot(AbstractINodeDiffList.java:296) at org.apache.hadoop.hdfs.server.namenode.INodeFile.recordModification(INodeFile.java:305) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.finalizeINodeFileUnderConstruction(FSNamesystem.java:4202) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.closeFileCommitBlocks(FSNamesystem.java:4419) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchronization(FSNamesystem.java:4383) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.commitBlockSynchronization(NameNodeRpcServer.java:699) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.commitBlockSynchronization(DatanodeProtocolServerSideTranslatorPB.java:270) at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:28073) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2053) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2047) {panel} > Block and lease recovery failure caused by snapshot issue > --------------------------------------------------------- > > Key: HDFS-7809 > URL: https://issues.apache.org/jira/browse/HDFS-7809 > Project: Hadoop HDFS > Issue Type: Bug > Affects Versions: 2.5.0 > Reporter: Kihwal Lee > Priority: Critical > > On a cluster running 2.5, we have observed a decommissioning failure due to a > file that had been under construction for 3 days. It turned out that the > file was abandoned and a lease recovery was carried out by the name node 3 > days ago. > The block recovery failed because the name node threw a quota exception while > serving {{commitBlockSynchronization()}}. After this failure, no further > attempt for recovery was made, leaving the file in under-construction state > forever. > Furthermore, the nature of the recovery failure is very strange. Even though > *snapshot was never used* in the cluster, it was trying to record the diff > and that required incrementing {{nsquota}} by 1. The user happened to ran out > of his {{nsquota}} at that time, so it failed and caused > {{commitBlockSynchronization()}} to fail. We do see quota discrepancies > occasionally. Probably those were caused by something like this all along? > Few observations: > - Lease recovery did not complete, yet didn't get retried. > - No snapshot was in use, but somehow it went through snapshot-related code > path. > - quota update during {{commitBlockSynchronization()}} should be done > unconditionally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)