[ https://issues.apache.org/jira/browse/HDFS-9476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265396#comment-15265396 ]
Masatake Iwasaki commented on HDFS-9476: ---------------------------------------- The issue was reproduced even after the 001 was applied. {noformat} 2016-04-30 22:09:02,681 [IPC Server handler 1 on 38857] INFO FSNamesystem.audit (FSNamesystem.java:logAuditMessage(6997)) - allowed=true ugi=iwasakims (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/1kb-multiple-checksum-blocks-64-16 dst=null perm=null proto=rpc 2016-04-30 22:09:02,753 [main] INFO hdfs.TestDFSUpgradeFromImage (TestDFSUpgradeFromImage.java:dfsOpenFileWithRetries(187)) - Open failed. 0 times. Retrying. ...(snip) 2016-04-30 22:09:32,048 [IPC Server handler 4 on 38857] INFO FSNamesystem.audit (FSNamesystem.java:logAuditMessage(6997)) - allowed=true ugi=iwasakims (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/1kb-multiple-checksum-blocks-64-16 dst=null perm=null proto=rpc 2016-04-30 22:09:32,056 [main] INFO hdfs.TestDFSUpgradeFromImage (TestDFSUpgradeFromImage.java:dfsOpenFileWithRetries(187)) - Open failed. 29 times. Retrying. {noformat} The test seems to fail if lease recovery happens (due to client's request) before the datanode's storage report is processed. This looks like not a test problem but real race. {noformat} 2016-04-30 22:09:02,537 [IPC Server handler 2 on 38857] INFO FSNamesystem.audit (FSNamesystem.java:logAuditMessage(6997)) - allowed=true ugi=iwasakims (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/1kb-multiple-checksum-blocks-64-16 dst=null perm=null proto=rpc 2016-04-30 22:09:02,544 [IPC Server handler 4 on 38857] INFO namenode.FSNamesystem (FSNamesystem.java:recoverLeaseInternal(2365)) - recoverLease: [Lease. Holder: DFSClient_8256078, pending creates: 13], src=/1kb-multiple-checksum-blocks-64-16 from client DFSClient_8256078 2016-04-30 22:09:02,544 [IPC Server handler 4 on 38857] INFO namenode.FSNamesystem (FSNamesystem.java:internalReleaseLease(3096)) - Recovering [Lease. Holder: DFSClient_8256078, pending creates: 13], src=/1kb-multiple-checksum-blocks-64-16 2016-04-30 22:09:02,545 [IPC Server handler 4 on 38857] WARN BlockStateChange (BlockUnderConstructionFeature.java:initializeBlockRecovery(205)) - BLOCK* BlockUnderConstructionFeature.initializeBlockRecovery: No blocks found, lease removed. 2016-04-30 22:09:02,545 [IPC Server handler 4 on 38857] WARN hdfs.StateChange (FSNamesystem.java:internalReleaseLease(3220)) - DIR* NameSystem.internalReleaseLease: File /1kb-multiple-checksum-blocks-64-16 has not been closed. Lease recovery is in progress. RecoveryId = 1031 for block blk_7162739548153522810_1020 2016-04-30 22:09:02,545 [Block report processor] INFO blockmanagement.BlockManager (BlockManager.java:processReport(2174)) - Processing first storage report for DS-a78b43f5-967a-47e3-afe1-68bb2f73a7eb from datanode e6f67da5-812b-402b-b5ac-dad5deab2666 {noformat} > TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage occasionally fail > --------------------------------------------------------------------- > > Key: HDFS-9476 > URL: https://issues.apache.org/jira/browse/HDFS-9476 > Project: Hadoop HDFS > Issue Type: Bug > Reporter: Wei-Chiu Chuang > Assignee: Akira AJISAKA > Attachments: HDFS-9476.01.patch > > > This test occasionally fail. For example, the most recent one is: > https://builds.apache.org/job/Hadoop-Hdfs-trunk/2587/ > Error Message > {noformat} > Cannot obtain block length for > LocatedBlock{BP-1371507683-67.195.81.153-1448798439809:blk_7162739548153522810_1020; > getBlockSize()=1024; corrupt=false; offset=0; > locs=[DatanodeInfoWithStorage[127.0.0.1:33080,DS-c5eaf2b4-2ee6-419d-a8a0-44a5df5ef9a1,DISK]]} > {noformat} > Stacktrace > {noformat} > java.io.IOException: Cannot obtain block length for > LocatedBlock{BP-1371507683-67.195.81.153-1448798439809:blk_7162739548153522810_1020; > getBlockSize()=1024; corrupt=false; offset=0; > locs=[DatanodeInfoWithStorage[127.0.0.1:33080,DS-c5eaf2b4-2ee6-419d-a8a0-44a5df5ef9a1,DISK]]} > at > org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:399) > at > org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:343) > at > org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:275) > at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:265) > at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1046) > at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1011) > at > org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.dfsOpenFileWithRetries(TestDFSUpgradeFromImage.java:177) > at > org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.verifyDir(TestDFSUpgradeFromImage.java:213) > at > org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.verifyFileSystem(TestDFSUpgradeFromImage.java:228) > at > org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.upgradeAndVerify(TestDFSUpgradeFromImage.java:600) > at > org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.testUpgradeFromRel1BBWImage(TestDFSUpgradeFromImage.java:622) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org