[ https://issues.apache.org/jira/browse/HDFS-9580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wei-Chiu Chuang updated HDFS-9580: ---------------------------------- Description: The failure appeared in the trunk jenkins job. https://builds.apache.org/job/Hadoop-Hdfs-trunk/2646/ {noformat} Error Message Expected invalidate blocks to be the number of DNs expected:<3> but was:<2> Stacktrace java.lang.AssertionError: Expected invalidate blocks to be the number of DNs expected:<3> but was:<2> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork.testDatanodeReRegistration(TestComputeInvalidateWork.java:160) {noformat} I think there could be a race condition between creating a file and shutting down data nodes, which failed the test. {noformat} 2015-12-19 07:11:02,765 [PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=LAST_IN_PIPELINE, downstreams=0:[]] INFO datanode.DataNode (BlockReceiver.java:run(1404)) - PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=LAST_IN_PIPELINE, downstreams=0:[] terminating 2015-12-19 07:11:02,768 [PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=HAS_DOWNSTREAM_IN_PIPELINE] INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1431)) - src: /127.0.0.1:45655, dest: /127.0.0.1:54890, bytes: 134217728, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_147911011_935, offset: 0, srvID: 6a13ec05-e1c1-4086-8a4d-d5a09636afcd, blockid: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, duration: 954174423 2015-12-19 07:11:02,768 [PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=HAS_DOWNSTREAM_IN_PIPELINE] INFO datanode.DataNode (BlockReceiver.java:run(1404)) - PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 2015-12-19 07:11:02,772 [PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=HAS_DOWNSTREAM_IN_PIPELINE] INFO DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1431)) - src: /127.0.0.1:33252, dest: /127.0.0.1:54426, bytes: 134217728, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_147911011_935, offset: 0, srvID: d81751db-02a9-48fe-b697-77623048784b, blockid: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, duration: 957463510 2015-12-19 07:11:02,772 [PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=HAS_DOWNSTREAM_IN_PIPELINE] INFO datanode.DataNode (BlockReceiver.java:run(1404)) - PacketResponder: BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, type=HAS_DOWNSTREAM_IN_PIPELINE terminating 2015-12-19 07:11:02,782 [IPC Server handler 4 on 36404] INFO blockmanagement.BlockManager (BlockManager.java:checkBlocksProperlyReplicated(3871)) - BLOCK* blk_1073741825_1001 is not COMPLETE (ucState = COMMITTED, replication# = 0 < minimum = 1) in file /testRR 2015-12-19 07:11:02,783 [IPC Server handler 4 on 36404] INFO namenode.EditLogFileOutputStream (EditLogFileOutputStream.java:flushAndSync(200)) - Nothing to flush 2015-12-19 07:11:02,783 [IPC Server handler 4 on 36404] INFO namenode.EditLogFileOutputStream (EditLogFileOutputStream.java:flushAndSync(200)) - Nothing to flush 2015-12-19 07:11:03,190 [IPC Server handler 8 on 36404] INFO hdfs.StateChange (FSNamesystem.java:completeFile(2557)) - DIR* completeFile: /testRR is closed by DFSClient_NONMAPREDUCE_147911011_935 {noformat} was: The failure appeared in the trunk jenkins job. https://builds.apache.org/job/Hadoop-Hdfs-trunk/2646/ {noformat} Error Message Expected invalidate blocks to be the number of DNs expected:<3> but was:<2> Stacktrace java.lang.AssertionError: Expected invalidate blocks to be the number of DNs expected:<3> but was:<2> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork.testDatanodeReRegistration(TestComputeInvalidateWork.java:160) {noformat} I think there could be a race condition between creating a file and shutting down data nodes, which failed the test. > TestComputeInvalidateWork#testDatanodeReRegistration failed due to unexpected > number of invalidate blocks. > ---------------------------------------------------------------------------------------------------------- > > Key: HDFS-9580 > URL: https://issues.apache.org/jira/browse/HDFS-9580 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, namenode, test > Affects Versions: 3.0.0 > Environment: Jenkins > Reporter: Wei-Chiu Chuang > Assignee: Wei-Chiu Chuang > > The failure appeared in the trunk jenkins job. > https://builds.apache.org/job/Hadoop-Hdfs-trunk/2646/ > {noformat} > Error Message > Expected invalidate blocks to be the number of DNs expected:<3> but was:<2> > Stacktrace > java.lang.AssertionError: Expected invalidate blocks to be the number of DNs > expected:<3> but was:<2> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at > org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork.testDatanodeReRegistration(TestComputeInvalidateWork.java:160) > {noformat} > I think there could be a race condition between creating a file and shutting > down data nodes, which failed the test. > {noformat} > 2015-12-19 07:11:02,765 [PacketResponder: > BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, > type=LAST_IN_PIPELINE, downstreams=0:[]] INFO datanode.DataNode > (BlockReceiver.java:run(1404)) - PacketResponder: > BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, > type=LAST_IN_PIPELINE, downstreams=0:[] terminating > 2015-12-19 07:11:02,768 [PacketResponder: > BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, > type=HAS_DOWNSTREAM_IN_PIPELINE] INFO DataNode.clienttrace > (BlockReceiver.java:finalizeBlock(1431)) - src: /127.0.0.1:45655, dest: > /127.0.0.1:54890, bytes: 134217728, op: HDFS_WRITE, cliID: > DFSClient_NONMAPREDUCE_147911011_935, offset: 0, srvID: > 6a13ec05-e1c1-4086-8a4d-d5a09636afcd, blockid: > BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, duration: > 954174423 > 2015-12-19 07:11:02,768 [PacketResponder: > BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, > type=HAS_DOWNSTREAM_IN_PIPELINE] INFO datanode.DataNode > (BlockReceiver.java:run(1404)) - PacketResponder: > BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, > type=HAS_DOWNSTREAM_IN_PIPELINE terminating > 2015-12-19 07:11:02,772 [PacketResponder: > BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, > type=HAS_DOWNSTREAM_IN_PIPELINE] INFO DataNode.clienttrace > (BlockReceiver.java:finalizeBlock(1431)) - src: /127.0.0.1:33252, dest: > /127.0.0.1:54426, bytes: 134217728, op: HDFS_WRITE, cliID: > DFSClient_NONMAPREDUCE_147911011_935, offset: 0, srvID: > d81751db-02a9-48fe-b697-77623048784b, blockid: > BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, duration: > 957463510 > 2015-12-19 07:11:02,772 [PacketResponder: > BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, > type=HAS_DOWNSTREAM_IN_PIPELINE] INFO datanode.DataNode > (BlockReceiver.java:run(1404)) - PacketResponder: > BP-1551077294-67.195.81.149-1450509060247:blk_1073741825_1001, > type=HAS_DOWNSTREAM_IN_PIPELINE terminating > 2015-12-19 07:11:02,782 [IPC Server handler 4 on 36404] INFO > blockmanagement.BlockManager > (BlockManager.java:checkBlocksProperlyReplicated(3871)) - BLOCK* > blk_1073741825_1001 is not COMPLETE (ucState = COMMITTED, replication# = 0 < > minimum = 1) in file /testRR > 2015-12-19 07:11:02,783 [IPC Server handler 4 on 36404] INFO > namenode.EditLogFileOutputStream > (EditLogFileOutputStream.java:flushAndSync(200)) - Nothing to flush > 2015-12-19 07:11:02,783 [IPC Server handler 4 on 36404] INFO > namenode.EditLogFileOutputStream > (EditLogFileOutputStream.java:flushAndSync(200)) - Nothing to flush > 2015-12-19 07:11:03,190 [IPC Server handler 8 on 36404] INFO > hdfs.StateChange (FSNamesystem.java:completeFile(2557)) - DIR* completeFile: > /testRR is closed by DFSClient_NONMAPREDUCE_147911011_935 > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)