[jira] [Created] (HDFS-13515) NetUtils#connect should log remote address for NoRouteToHostException
Ted Yu created HDFS-13515: - Summary: NetUtils#connect should log remote address for NoRouteToHostException Key: HDFS-13515 URL: https://issues.apache.org/jira/browse/HDFS-13515 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ted Yu {code} hdfs.BlockReaderFactory: I/O error constructing remote block reader. java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2884) {code} In the above stack trace, the remote host was not logged. This makes troubleshooting a bit hard. NetUtils#connect should log remote address for NoRouteToHostException . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13511) Provide specialized exception when block length cannot be obtained
Ted Yu created HDFS-13511: - Summary: Provide specialized exception when block length cannot be obtained Key: HDFS-13511 URL: https://issues.apache.org/jira/browse/HDFS-13511 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ted Yu In downstream project, I saw the following code: {code} FSDataInputStream inputStream = hdfs.open(new Path(path)); ... if (options.getRecoverFailedOpen() && dfs != null && e.getMessage().toLowerCase() .startsWith("cannot obtain block length for")) { {code} The above tightly depends on the following in DFSInputStream#readBlockLength {code} throw new IOException("Cannot obtain block length for " + locatedblock); {code} The check based on string matching is brittle in production deployment. After discussing with [~ste...@apache.org], better approach is to introduce specialized IOException, e.g. CannotObtainBlockLengthException so that downstream project doesn't have to rely on string matching. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13206) IllegalStateException: Unable to finalize edits file
Ted Yu created HDFS-13206: - Summary: IllegalStateException: Unable to finalize edits file Key: HDFS-13206 URL: https://issues.apache.org/jira/browse/HDFS-13206 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ted Yu I noticed the following in hbase test output running against hadoop3: {code} 2018-02-28 18:40:18,491 ERROR [Time-limited test] namenode.JournalSet(402): Error: finalize log segment 1, 658 failed for (journal JournalAndStream(mgr=FileJournalManager(root=/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1), stream=null)) java.lang.IllegalStateException: Unable to finalize edits file /mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1/current/edits_inprogress_001 at org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:153) at org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:224) at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385) at org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:219) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1427) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:398) at org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.close(FSEditLogAsync.java:110) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1320) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1909) at org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70) at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1013) at org.apache.hadoop.hdfs.MiniDFSCluster.stopAndJoinNameNode(MiniDFSCluster.java:2047) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1987) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1958) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1951) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:767) at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1109) at org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport.stopCluster(TestFavoredNodeTableImport.java:71) {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-5012) replica.getGenerationStamp() may be >= recoveryId
[ https://issues.apache.org/jira/browse/HDFS-5012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-5012. -- Resolution: Cannot Reproduce > replica.getGenerationStamp() may be >= recoveryId > - > > Key: HDFS-5012 > URL: https://issues.apache.org/jira/browse/HDFS-5012 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.5-alpha >Reporter: Ted Yu > Attachments: testReplicationQueueFailover.txt > > > The following was first observed by [~jdcryans] in > TestReplicationQueueFailover running against 2.0.5-alpha: > {code} > 2013-07-16 17:14:33,340 ERROR [IPC Server handler 7 on 35081] > security.UserGroupInformation(1481): PriviledgedActionException as:ec2-user > (auth:SIMPLE) cause:java.io.IOException: THIS IS NOT SUPPOSED TO HAPPEN: > replica.getGenerationStamp() >= recoveryId = 1041, > block=blk_4297992342878601848_1041, replica=FinalizedReplica, > blk_4297992342878601848_1041, FINALIZED > getNumBytes() = 794 > getBytesOnDisk() = 794 > getVisibleLength()= 794 > getVolume() = > /home/ec2-user/jenkins/workspace/HBase-0.95-Hadoop-2/hbase-server/target/test-data/f2763e32-fe49-4988-ac94-eeca82431821/dfscluster_643a635e-4e39-4aa5-974c-25e01db16ff7/dfs/data/data3/current > getBlockFile()= > /home/ec2-user/jenkins/workspace/HBase-0.95-Hadoop-2/hbase-server/target/test-data/f2763e32-fe49-4988-ac94-eeca82431821/dfscluster_643a635e-4e39-4aa5-974c-25e01db16ff7/dfs/data/data3/current/BP-1477359609-10.197.55.49-1373994849464/current/finalized/blk_4297992342878601848 > unlinked =false > 2013-07-16 17:14:33,341 WARN > [org.apache.hadoop.hdfs.server.datanode.DataNode$2@64a1fcba] > datanode.DataNode(1894): Failed to obtain replica info for block > (=BP-1477359609-10.197.55.49-1373994849464:blk_4297992342878601848_1041) from > datanode (=127.0.0.1:47006) > java.io.IOException: THIS IS NOT SUPPOSED TO HAPPEN: > replica.getGenerationStamp() >= recoveryId = 1041, > block=blk_4297992342878601848_1041, replica=FinalizedReplica, > blk_4297992342878601848_1041, FINALIZED > getNumBytes() = 794 > getBytesOnDisk() = 794 > getVisibleLength()= 794 > getVolume() = > /home/ec2-user/jenkins/workspace/HBase-0.95-Hadoop-2/hbase-server/target/test-data/f2763e32-fe49-4988-ac94-eeca82431821/dfscluster_643a635e-4e39-4aa5-974c-25e01db16ff7/dfs/data/data3/current > getBlockFile()= > /home/ec2-user/jenkins/workspace/HBase-0.95-Hadoop-2/hbase-server/target/test-data/f2763e32-fe49-4988-ac94-eeca82431821/dfscluster_643a635e-4e39-4aa5-974c-25e01db16ff7/dfs/data/data3/current/BP-1477359609-10.197.55.49-1373994849464/current/finalized/blk_4297992342878601848 > unlinked =false > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-4796) Port HDFS-4721 'Speed up lease/block recovery when DN fails and a block goes into recovery' to branch 1
[ https://issues.apache.org/jira/browse/HDFS-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-4796. -- Resolution: Won't Fix > Port HDFS-4721 'Speed up lease/block recovery when DN fails and a block goes > into recovery' to branch 1 > --- > > Key: HDFS-4796 > URL: https://issues.apache.org/jira/browse/HDFS-4796 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ted Yu > > This was observed while doing HBase WAL recovery. HBase uses append to write > to its write ahead log. So initially the pipeline is setup as > DN1 --> DN2 --> DN3 > This WAL needs to be read when DN1 fails since it houses the HBase > regionserver for the WAL. > HBase first recovers the lease on the WAL file. During recovery, we choose > DN1 as the primary DN to do the recovery even though DN1 has failed and is > not heartbeating any more. > To speedup lease/block recovery, we always choose the datanode with the most > recent heartbeat. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-5834) TestCheckpoint#testCheckpoint may fail due to Bad value assertion
[ https://issues.apache.org/jira/browse/HDFS-5834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-5834. -- Resolution: Cannot Reproduce > TestCheckpoint#testCheckpoint may fail due to Bad value assertion > - > > Key: HDFS-5834 > URL: https://issues.apache.org/jira/browse/HDFS-5834 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > I saw the following when running test suite on Linux: > {code} > testCheckpoint(org.apache.hadoop.hdfs.server.namenode.TestCheckpoint) Time > elapsed: 3.058 sec <<< FAILURE! > java.lang.AssertionError: Bad value for metric GetImageNumOps > Expected: gt(0) > got: <0L> > at org.junit.Assert.assertThat(Assert.java:780) > at > org.apache.hadoop.test.MetricsAsserts.assertCounterGt(MetricsAsserts.java:318) > at > org.apache.hadoop.hdfs.server.namenode.TestCheckpoint.testCheckpoint(TestCheckpoint.java:1058) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-5718) TestHttpsFileSystem intermittently fails with Port in use error
[ https://issues.apache.org/jira/browse/HDFS-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-5718. -- Resolution: Cannot Reproduce > TestHttpsFileSystem intermittently fails with Port in use error > --- > > Key: HDFS-5718 > URL: https://issues.apache.org/jira/browse/HDFS-5718 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From > https://builds.apache.org/job/Hadoop-Hdfs-trunk/1634/testReport/junit/org.apache.hadoop.hdfs.web/TestHttpsFileSystem/org_apache_hadoop_hdfs_web_TestHttpsFileSystem/ > : > {code} > java.net.BindException: Port in use: localhost:50475 > at java.net.PlainSocketImpl.socketBind(Native Method) > at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383) > at java.net.ServerSocket.bind(ServerSocket.java:328) > at java.net.ServerSocket.(ServerSocket.java:194) > at javax.net.ssl.SSLServerSocket.(SSLServerSocket.java:106) > at > com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.(SSLServerSocketImpl.java:108) > at > com.sun.net.ssl.internal.ssl.SSLServerSocketFactoryImpl.createServerSocket(SSLServerSocketFactoryImpl.java:72) > at > org.mortbay.jetty.security.SslSocketConnector.newServerSocket(SslSocketConnector.java:478) > at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73) > at org.apache.hadoop.http.HttpServer.openListeners(HttpServer.java:973) > at org.apache.hadoop.http.HttpServer.start(HttpServer.java:914) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:412) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:769) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:315) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1846) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1746) > at > org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1203) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:673) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:342) > at > org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:323) > at > org.apache.hadoop.hdfs.web.TestHttpsFileSystem.setUp(TestHttpsFileSystem.java:64) > {code} > This could have been caused by concurrent test(s). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-5831) TestAuditLogs#testAuditAllowedStat sometimes fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-5831. -- Resolution: Cannot Reproduce > TestAuditLogs#testAuditAllowedStat sometimes fails in trunk > --- > > Key: HDFS-5831 > URL: https://issues.apache.org/jira/browse/HDFS-5831 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu > Attachments: > 5831-org.apache.hadoop.hdfs.server.namenode.TestAuditLogs-output.txt > > > Running TestAuditLogs on Linux, I got: > {code} > testAuditAllowedStat[1](org.apache.hadoop.hdfs.server.namenode.TestAuditLogs) > Time elapsed: 6.677 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:92) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNotNull(Assert.java:526) > at org.junit.Assert.assertNotNull(Assert.java:537) > at > org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.verifyAuditLogsRepeat(TestAuditLogs.java:312) > at > org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.verifyAuditLogs(TestAuditLogs.java:295) > at > org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.testAuditAllowedStat(TestAuditLogs.java:163) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12599) Move DataNodeTestUtils.mockDatanodeBlkPinning into mock test util class
Ted Yu created HDFS-12599: - Summary: Move DataNodeTestUtils.mockDatanodeBlkPinning into mock test util class Key: HDFS-12599 URL: https://issues.apache.org/jira/browse/HDFS-12599 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu HDFS-11164 introduced {{DataNodeTestUtils.mockDatanodeBlkPinning}} which brought dependency on mockito back into DataNodeTestUtils Downstream, this resulted in: {code} java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer at org.apache.hadoop.hdfs.MiniDFSCluster.shouldWait(MiniDFSCluster.java:2668) at org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:2564) at org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:2607) at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1667) at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:874) at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:769) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:661) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1075) at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:953) {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-9169) TestNativeAzureFileSystemOperationsMocked fails in trunk
Ted Yu created HDFS-9169: Summary: TestNativeAzureFileSystemOperationsMocked fails in trunk Key: HDFS-9169 URL: https://issues.apache.org/jira/browse/HDFS-9169 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor When working on HDFS-6264, QA bot reported the following: {code} testGlobStatusFilterWithMultiplePathWildcardsAndNonTrivialFilter(org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked) Time elapsed: 0.02 sec <<< ERROR! java.lang.NullPointerException: null at org.apache.hadoop.fs.Globber.glob(Globber.java:145) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1688) at org.apache.hadoop.fs.FSMainOperationsBaseTest.testGlobStatusFilterWithMultiplePathWildcardsAndNonTrivialFilter(FSMainOp {code} On hadoop trunk branch, the above can be reproduced without any patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-5897) TestNNWithQJM#testNewNamenodeTakesOverWriter occasionally fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-5897. -- Resolution: Cannot Reproduce > TestNNWithQJM#testNewNamenodeTakesOverWriter occasionally fails in trunk > > > Key: HDFS-5897 > URL: https://issues.apache.org/jira/browse/HDFS-5897 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu > Attachments: 5897-output.html > > > From > https://builds.apache.org/job/Hadoop-Hdfs-trunk/1665/testReport/junit/org.apache.hadoop.hdfs.qjournal/TestNNWithQJM/testNewNamenodeTakesOverWriter/ > : > {code} > java.lang.Exception: test timed out after 3 milliseconds > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:129) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) > at java.io.BufferedInputStream.read(BufferedInputStream.java:317) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379) > at > org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:412) > at > org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:401) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > {code} > I saw: > {code} > 2014-02-06 11:38:37,970 ERROR namenode.EditLogInputStream > (RedundantEditLogInputStream.java:nextOp(221)) - Got error reading edit log > input stream > http://localhost:40509/getJournal?jid=myjournal&segmentTxId=3&storageInfo=-51%3A1571339494%3A0%3AtestClusterID; > failing over to edit log > http://localhost:56244/getJournal?jid=myjournal&segmentTxId=3&storageInfo=-51%3A1571339494%3A0%3AtestClusterID > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: > got premature end-of-file at txid 0; expected file to go up to 4 > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:83) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:140) > at > org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) > at > org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:83) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:167) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:120) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:708) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:606) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:263) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:874) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:634) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:446) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:502) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:658) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:643) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1291) > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:939) > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:824) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:678) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:359) > at > org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:340) > at > org.apache.hadoop.hdfs.qjournal.TestNNWithQJM.testNewNamenodeTakesOverWriter(TestNNWithQJM.java:145) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.
[jira] [Resolved] (HDFS-6037) TestIncrementalBlockReports#testReplaceReceivedBlock fails occasionally in trunk
[ https://issues.apache.org/jira/browse/HDFS-6037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-6037. -- Resolution: Cannot Reproduce > TestIncrementalBlockReports#testReplaceReceivedBlock fails occasionally in > trunk > > > Key: HDFS-6037 > URL: https://issues.apache.org/jira/browse/HDFS-6037 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu > > From > https://builds.apache.org/job/Hadoop-Hdfs-trunk/1688/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestIncrementalBlockReports/testReplaceReceivedBlock/ > : > {code} > datanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted( > , > , > > ); > Wanted 1 time: > -> at > org.apache.hadoop.hdfs.server.datanode.TestIncrementalBlockReports.testReplaceReceivedBlock(TestIncrementalBlockReports.java:198) > But was 2 times. Undesired invocation: > -> at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.reportReceivedDeletedBlocks(BPServiceActor.java:303) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-6081) TestRetryCacheWithHA#testCreateSymlink occasionally fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-6081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-6081. -- Resolution: Cannot Reproduce > TestRetryCacheWithHA#testCreateSymlink occasionally fails in trunk > -- > > Key: HDFS-6081 > URL: https://issues.apache.org/jira/browse/HDFS-6081 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu > > From > https://builds.apache.org/job/Hadoop-Hdfs-trunk/1696/testReport/junit/org.apache.hadoop.hdfs.server.namenode.ha/TestRetryCacheWithHA/testCreateSymlink/ > : > {code} > 2014-03-09 13:18:47,515 WARN security.UserGroupInformation > (UserGroupInformation.java:doAs(1600)) - PriviledgedActionException > as:jenkins (auth:SIMPLE) cause:java.io.IOException: failed to create link > /testlink either because the filename is invalid or the file exists > 2014-03-09 13:18:47,515 INFO ipc.Server (Server.java:run(2093)) - IPC Server > handler 0 on 39303, call > org.apache.hadoop.hdfs.protocol.ClientProtocol.createSymlink from > 127.0.0.1:32909 Call#682 Retry#1: error: java.io.IOException: failed to > create link /testlink either because the filename is invalid or the file > exists > java.io.IOException: failed to create link /testlink either because the > filename is invalid or the file exists > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlinkInt(FSNamesystem.java:2053) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlink(FSNamesystem.java:2023) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.createSymlink(NameNodeRpcServer.java:965) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.createSymlink(ClientNamenodeProtocolServerSideTranslatorPB.java:844) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:605) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2071) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2067) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1597) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2065) > 2014-03-09 13:18:47,522 INFO blockmanagement.BlockManager > (BlockManager.java:processMisReplicatesAsync(2475)) - Total number of blocks > = 1 > 2014-03-09 13:18:47,523 INFO blockmanagement.BlockManager > (BlockManager.java:processMisReplicatesAsync(2476)) - Number of invalid > blocks = 0 > 2014-03-09 13:18:47,523 INFO blockmanagement.BlockManager > (BlockManager.java:processMisReplicatesAsync(2477)) - Number of > under-replicated blocks = 0 > 2014-03-09 13:18:47,523 INFO ha.TestRetryCacheWithHA > (TestRetryCacheWithHA.java:run(1162)) - Got Exception while calling > createSymlink > org.apache.hadoop.ipc.RemoteException(java.io.IOException): failed to create > link /testlink either because the filename is invalid or the file exists > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlinkInt(FSNamesystem.java:2053) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlink(FSNamesystem.java:2023) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.createSymlink(NameNodeRpcServer.java:965) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.createSymlink(ClientNamenodeProtocolServerSideTranslatorPB.java:844) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:605) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2071) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2067) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1597) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2065) > at org.apache.hadoop.ipc.Client.call(Client.java:1409) > at org.apache.hadoop.ipc.Client.call(Client.java:1362) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) >
[jira] [Resolved] (HDFS-6501) TestCrcCorruption#testCorruptionDuringWrt sometimes fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-6501. -- Resolution: Cannot Reproduce > TestCrcCorruption#testCorruptionDuringWrt sometimes fails in trunk > -- > > Key: HDFS-6501 > URL: https://issues.apache.org/jira/browse/HDFS-6501 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1767/ : > {code} > REGRESSION: org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt > Error Message: > test timed out after 5 milliseconds > Stack Trace: > java.lang.Exception: test timed out after 5 milliseconds > at java.lang.Object.wait(Native Method) > at > org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2024) > at > org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2008) > at > org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2107) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70) > at > org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:98) > at > org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt(TestCrcCorruption.java:133) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-6726) TestNamenodeCapacityReport fails intermittently
[ https://issues.apache.org/jira/browse/HDFS-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-6726. -- Resolution: Cannot Reproduce > TestNamenodeCapacityReport fails intermittently > --- > > Key: HDFS-6726 > URL: https://issues.apache.org/jira/browse/HDFS-6726 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From > https://builds.apache.org/job/Hadoop-Hdfs-trunk/1812/testReport/junit/org.apache.hadoop.hdfs.server.namenode/TestNamenodeCapacityReport/testXceiverCount/ > : > {code} > java.io.IOException: Unable to close file because the last block does not > have enough number of replicas. > at > org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2141) > at > org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2109) > at > org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport.testXceiverCount(TestNamenodeCapacityReport.java:281) > {code} > There were multiple occurrences of 'Broken pipe', 'Connection reset by peer' > and 'Premature EOF from inputStream' exceptions in test output -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-6177) TestHttpFSServer fails occasionally in trunk
[ https://issues.apache.org/jira/browse/HDFS-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-6177. -- Resolution: Cannot Reproduce > TestHttpFSServer fails occasionally in trunk > > > Key: HDFS-6177 > URL: https://issues.apache.org/jira/browse/HDFS-6177 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From https://builds.apache.org/job/Hadoop-hdfs-trunk/1716/consoleFull : > {code} > Running org.apache.hadoop.fs.http.server.TestHttpFSServer > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.424 sec <<< > FAILURE! - in org.apache.hadoop.fs.http.server.TestHttpFSServer > testDelegationTokenOperations(org.apache.hadoop.fs.http.server.TestHttpFSServer) > Time elapsed: 0.559 sec <<< FAILURE! > java.lang.AssertionError: expected:<401> but was:<403> > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.failNotEquals(Assert.java:647) > at org.junit.Assert.assertEquals(Assert.java:128) > at org.junit.Assert.assertEquals(Assert.java:472) > at org.junit.Assert.assertEquals(Assert.java:456) > at > org.apache.hadoop.fs.http.server.TestHttpFSServer.testDelegationTokenOperations(TestHttpFSServer.java:352) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7083) TestDecommission#testIncludeByRegistrationName sometimes fails
[ https://issues.apache.org/jira/browse/HDFS-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-7083. -- Resolution: Cannot Reproduce > TestDecommission#testIncludeByRegistrationName sometimes fails > -- > > Key: HDFS-7083 > URL: https://issues.apache.org/jira/browse/HDFS-7083 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1874/ : > {code} > REGRESSION: > org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName > Error Message: > test timed out after 36 milliseconds > Stack Trace: > java.lang.Exception: test timed out after 36 milliseconds > at java.lang.Thread.sleep(Native Method) > at > org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName(TestDecommission.java:957) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7102) Null dereference in PacketReceiver#receiveNextPacket()
[ https://issues.apache.org/jira/browse/HDFS-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-7102. -- Resolution: Later > Null dereference in PacketReceiver#receiveNextPacket() > -- > > Key: HDFS-7102 > URL: https://issues.apache.org/jira/browse/HDFS-7102 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ted Yu >Priority: Minor > > {code} > public void receiveNextPacket(ReadableByteChannel in) throws IOException { > doRead(in, null); > {code} > doRead() would pass null as second parameter to (line 134): > {code} > doReadFully(ch, in, curPacketBuf); > {code} > which dereferences it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7756) DatanodeInfoWithStorage should be tagged Private
Ted Yu created HDFS-7756: Summary: DatanodeInfoWithStorage should be tagged Private Key: HDFS-7756 URL: https://issues.apache.org/jira/browse/HDFS-7756 Project: Hadoop HDFS Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu This is related to HDFS-7647 DatanodeInfoWithStorage was introduced in org.apache.hadoop.hdfs.server.protocol package whereas its base class, DatanodeInfo, is in org.apache.hadoop.hdfs.protocol DatanodeInfo is tagged @InterfaceAudience.Private DatanodeInfoWithStorage should have the same tag. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7220) TestDataNodeMetrics fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-7220. -- Resolution: Cannot Reproduce > TestDataNodeMetrics fails in trunk > -- > > Key: HDFS-7220 > URL: https://issues.apache.org/jira/browse/HDFS-7220 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From > https://builds.apache.org/job/Hadoop-Hdfs-trunk/1896/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeMetrics/testSendDataPacketMetrics/ > : > {code} > java.lang.NoClassDefFoundError: > org/apache/hadoop/util/IntrusiveCollection$IntrusiveIterator > at > org.apache.hadoop.util.IntrusiveCollection.iterator(IntrusiveCollection.java:213) > at > org.apache.hadoop.util.IntrusiveCollection.clear(IntrusiveCollection.java:368) > at > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.clearPendingCachingCommands(DatanodeManager.java:1590) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1262) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.close(FSNamesystem.java:1590) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.stopCommonServices(NameNode.java:658) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:823) > at > org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1717) > at > org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1696) > at > org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics.testSendDataPacketMetrics(TestDataNodeMetrics.java:94) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7267) TestBalancer#testUnknownDatanode occasionally fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-7267. -- Resolution: Cannot Reproduce > TestBalancer#testUnknownDatanode occasionally fails in trunk > > > Key: HDFS-7267 > URL: https://issues.apache.org/jira/browse/HDFS-7267 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > In build #1907 (https://builds.apache.org/job/Hadoop-Hdfs-trunk/1907/): > {code} > REGRESSION: > org.apache.hadoop.hdfs.server.balancer.TestBalancer.testUnknownDatanode > Error Message: > expected:<0> but was:<-3> > Stack Trace: > java.lang.AssertionError: expected:<0> but was:<-3> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.hdfs.server.balancer.TestBalancer.testUnknownDatanode(TestBalancer.java:737) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7571) TestCacheDirectives#testExceedsCapacity sometimes fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-7571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-7571. -- Resolution: Cannot Reproduce > TestCacheDirectives#testExceedsCapacity sometimes fails in trunk > > > Key: HDFS-7571 > URL: https://issues.apache.org/jira/browse/HDFS-7571 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1985/ : > {code} > REGRESSION: > org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.testExceedsCapacity > Error Message: > Pending cached list of 127.0.0.1:47332 is not empty, [{blockId=1073741841, > replication=1, mark=true}] > Stack Trace: > java.lang.AssertionError: Pending cached list of 127.0.0.1:47332 is not > empty, [{blockId=1073741841, replication=1, mark=true}] > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.checkPendingCachedEmpty(TestCacheDirectives.java:1420) > at > org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.testExceedsCapacity(TestCacheDirectives.java:1443) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7576) TestPipelinesFailover#testFailoverRightBeforeCommitSynchronization sometimes fails in Java 8 build
[ https://issues.apache.org/jira/browse/HDFS-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-7576. -- Resolution: Cannot Reproduce > TestPipelinesFailover#testFailoverRightBeforeCommitSynchronization sometimes > fails in Java 8 build > -- > > Key: HDFS-7576 > URL: https://issues.apache.org/jira/browse/HDFS-7576 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/54/ : > {code} > REGRESSION: > org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization > Error Message: > test timed out after 3 milliseconds > Stack Trace: > java.lang.Exception: test timed out after 3 milliseconds > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) > at > org.apache.hadoop.test.GenericTestUtils$DelayAnswer.waitForCall(GenericTestUtils.java:226) > at > org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization(TestPipelinesFailover.java:386) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7289) TestDFSUpgradeWithHA sometimes fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-7289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-7289. -- Resolution: Cannot Reproduce > TestDFSUpgradeWithHA sometimes fails in trunk > - > > Key: HDFS-7289 > URL: https://issues.apache.org/jira/browse/HDFS-7289 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > Labels: ha > > From trunk build #1912: > {code} > REGRESSION: > org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA.testFinalizeFromSecondNameNodeWithJournalNodes > Error Message: > java.lang.RuntimeException: java.net.SocketTimeoutException: Read timed out > Stack Trace: > java.io.IOException: java.lang.RuntimeException: > java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:129) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) > at java.io.BufferedInputStream.read(BufferedInputStream.java:317) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:698) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:641) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1218) > at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379) > at > org.apache.hadoop.hdfs.server.namenode.TransferFsImage.doGetUrl(TransferFsImage.java:410) > at > org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:395) > at > org.apache.hadoop.hdfs.server.namenode.TransferFsImage.downloadImageToStorage(TransferFsImage.java:114) > at > org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.doRun(BootstrapStandby.java:213) > at > org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.access$000(BootstrapStandby.java:69) > at > org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby$1.run(BootstrapStandby.java:107) > at > org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby$1.run(BootstrapStandby.java:103) > at > org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:414) > at > org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:103) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:315) > at > org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA.testFinalizeFromSecondNameNodeWithJournalNodes(TestDFSUpgradeWithHA.java:493) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7311) TestLeaseRecovery2 sometimes fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-7311. -- Resolution: Cannot Reproduce > TestLeaseRecovery2 sometimes fails in trunk > --- > > Key: HDFS-7311 > URL: https://issues.apache.org/jira/browse/HDFS-7311 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1917/ : > {code} > REGRESSION: org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery > Error Message: > Call From asf909.gq1.ygridcore.net/67.195.81.153 to localhost:55061 failed on > connection exception: java.net.ConnectException: Connection refused; For more > details see: http://wiki.apache.org/hadoop/ConnectionRefused > Stack Trace: > java.net.ConnectException: Call From asf909.gq1.ygridcore.net/67.195.81.153 > to localhost:55061 failed on connection exception: java.net.ConnectException: > Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) > at > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) > at > org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705) > at > org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521) > at org.apache.hadoop.ipc.Client.call(Client.java:1438) > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) > at com.sun.proxy.$Proxy19.create(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:295) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101) > at com.sun.proxy.$Proxy20.create(Unknown Source) > at > org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1694) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1654) > at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1579) > at > org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:397) > at > org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:393) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:393) > at > org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:337) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908) > at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889) > at > org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery(TestLeaseRecovery2.java:276) > FAILED: > org.apache.hadoop.hdfs.TestLeaseRecovery2.org.apache.hadoop.hdfs.TestLeaseRecovery2 > Error Message: > Test resulted in an unexpected exit > Stack Trace: > java.lang.AssertionError: Test resulted in an unexpected exit > at > org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1709) > at > org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1696) > at > org.apache.hadoop.hdfs.TestLeaseRecovery2.tearDown(TestLeaseRecovery2.java:105) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7464) TestDFSAdminWithHA#testRefreshSuperUserGroupsConfiguration fails against Java 8
[ https://issues.apache.org/jira/browse/HDFS-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-7464. -- Resolution: Cannot Reproduce > TestDFSAdminWithHA#testRefreshSuperUserGroupsConfiguration fails against Java > 8 > --- > > Key: HDFS-7464 > URL: https://issues.apache.org/jira/browse/HDFS-7464 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/23/ : > {code} > REGRESSION: > org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA.testRefreshSuperUserGroupsConfiguration > Error Message: > refreshSuperUserGroupsConfiguration: End of File Exception between local host > is: "asf908.gq1.ygridcore.net/67.195.81.152"; destination host is: > "localhost":12700; : java.io.EOFException; For more details see: > http://wiki.apache.org/hadoop/EOFException expected:<0> but was:<-1> > Stack Trace: > java.lang.AssertionError: refreshSuperUserGroupsConfiguration: End of File > Exception between local host is: "asf908.gq1.ygridcore.net/67.195.81.152"; > destination host is: "localhost":12700; : java.io.EOFException; For more > details see: http://wiki.apache.org/hadoop/EOFException expected:<0> but > was:<-1> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at > org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA.testRefreshSuperUserGroupsConfiguration(TestDFSAdminWithHA.java:228) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7422) TestEncryptionZonesWithKMS fails against Java 8
[ https://issues.apache.org/jira/browse/HDFS-7422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-7422. -- Resolution: Cannot Reproduce > TestEncryptionZonesWithKMS fails against Java 8 > --- > > Key: HDFS-7422 > URL: https://issues.apache.org/jira/browse/HDFS-7422 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/12/ : > {code} > REGRESSION: > org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS.testReadWriteUsingWebHdfs > Error Message: > Stream closed. > Stack Trace: > java.io.IOException: Stream closed. > at sun.reflect.GeneratedConstructorAccessor58.newInstance(Unknown > Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:385) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:91) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:656) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:622) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$UnresolvedUrlOpener.connect(WebHdfsFileSystem.java:1204) > at > org.apache.hadoop.hdfs.web.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:120) > at > org.apache.hadoop.hdfs.web.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:104) > at > org.apache.hadoop.hdfs.web.ByteRangeInputStream.(ByteRangeInputStream.java:89) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$OffsetUrlInputStream.(WebHdfsFileSystem.java:1261) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.open(WebHdfsFileSystem.java:1175) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) > at > org.apache.hadoop.hdfs.DFSTestUtil.verifyFilesEqual(DFSTestUtil.java:1399) > at > org.apache.hadoop.hdfs.TestEncryptionZones.testReadWriteUsingWebHdfs(TestEncryptionZones.java:634) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) > Caused by: org.apache.hadoop.ipc.RemoteException: Stream closed. > at > org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:165) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:353) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683) > a
[jira] [Created] (HDFS-7606) Missing null check in INodeFile#getBlocks()
Ted Yu created HDFS-7606: Summary: Missing null check in INodeFile#getBlocks() Key: HDFS-7606 URL: https://issues.apache.org/jira/browse/HDFS-7606 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} BlockInfo[] snapshotBlocks = diff == null ? getBlocks() : diff.getBlocks(); if(snapshotBlocks != null) return snapshotBlocks; // Blocks are not in the current snapshot // Find next snapshot with blocks present or return current file blocks snapshotBlocks = getDiffs().findLaterSnapshotBlocks(diff.getSnapshotId()); {code} If diff is null and snapshotBlocks is null, NullPointerException would result from the call to diff.getSnapshotId(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7493) removedDst should be checked against null in finally block of FSDirRenameOp#unprotectedRenameTo()
[ https://issues.apache.org/jira/browse/HDFS-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-7493. -- Resolution: Duplicate Dup of HDFS-7538 > removedDst should be checked against null in finally block of > FSDirRenameOp#unprotectedRenameTo() > - > > Key: HDFS-7493 > URL: https://issues.apache.org/jira/browse/HDFS-7493 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ted Yu >Priority: Minor > > {code} > removedDst = dstIIP.getLastINode(); > undoRemoveDst = true; > {code} > If removedDst is null, the following code in finally block may result in NPE: > {code} > if (dstParent.isDirectory() && > dstParent.asDirectory().isWithSnapshot()) { > dstParent.asDirectory().undoRename4DstParent(removedDst, > dstIIP.getLatestSnapshotId()); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7576) TestPipelinesFailover#testFailoverRightBeforeCommitSynchronization sometimes fails in Java 8 build
Ted Yu created HDFS-7576: Summary: TestPipelinesFailover#testFailoverRightBeforeCommitSynchronization sometimes fails in Java 8 build Key: HDFS-7576 URL: https://issues.apache.org/jira/browse/HDFS-7576 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/54/ : {code} REGRESSION: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization Error Message: test timed out after 3 milliseconds Stack Trace: java.lang.Exception: test timed out after 3 milliseconds at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) at org.apache.hadoop.test.GenericTestUtils$DelayAnswer.waitForCall(GenericTestUtils.java:226) at org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization(TestPipelinesFailover.java:386) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7571) TestCacheDirectives#testExceedsCapacity sometimes fails in trunk
Ted Yu created HDFS-7571: Summary: TestCacheDirectives#testExceedsCapacity sometimes fails in trunk Key: HDFS-7571 URL: https://issues.apache.org/jira/browse/HDFS-7571 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1985/ : {code} REGRESSION: org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.testExceedsCapacity Error Message: Pending cached list of 127.0.0.1:47332 is not empty, [{blockId=1073741841, replication=1, mark=true}] Stack Trace: java.lang.AssertionError: Pending cached list of 127.0.0.1:47332 is not empty, [{blockId=1073741841, replication=1, mark=true}] at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.checkPendingCachedEmpty(TestCacheDirectives.java:1420) at org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.testExceedsCapacity(TestCacheDirectives.java:1443) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HDFS-7567) Potential null dereference in FSEditLogLoader#applyEditLogOp()
[ https://issues.apache.org/jira/browse/HDFS-7567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HDFS-7567: -- Assignee: Ted Yu > Potential null dereference in FSEditLogLoader#applyEditLogOp() > -- > > Key: HDFS-7567 > URL: https://issues.apache.org/jira/browse/HDFS-7567 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > > {code} > INodeFile oldFile = INodeFile.valueOf(iip.getLastINode(), path, true); > if (oldFile != null && addCloseOp.overwrite) { > ... > INodeFile newFile = oldFile; > ... > // Update the salient file attributes. > newFile.setAccessTime(addCloseOp.atime, Snapshot.CURRENT_STATE_ID); > newFile.setModificationTime(addCloseOp.mtime, > Snapshot.CURRENT_STATE_ID); > {code} > The last two lines are not protected by null check. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7567) Potential null dereference in FSEditLogLoader#applyEditLogOp()
Ted Yu created HDFS-7567: Summary: Potential null dereference in FSEditLogLoader#applyEditLogOp() Key: HDFS-7567 URL: https://issues.apache.org/jira/browse/HDFS-7567 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} INodeFile oldFile = INodeFile.valueOf(iip.getLastINode(), path, true); if (oldFile != null && addCloseOp.overwrite) { ... INodeFile newFile = oldFile; ... // Update the salient file attributes. newFile.setAccessTime(addCloseOp.atime, Snapshot.CURRENT_STATE_ID); newFile.setModificationTime(addCloseOp.mtime, Snapshot.CURRENT_STATE_ID); {code} The last two lines are not protected by null check. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7538) removedDst should be checked against null in the finally block of FSDirRenameOp#unprotectedRenameTo()
Ted Yu created HDFS-7538: Summary: removedDst should be checked against null in the finally block of FSDirRenameOp#unprotectedRenameTo() Key: HDFS-7538 URL: https://issues.apache.org/jira/browse/HDFS-7538 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} if (removedDst != null) { undoRemoveDst = false; ... if (undoRemoveDst) { // Rename failed - restore dst if (dstParent.isDirectory() && dstParent.asDirectory().isWithSnapshot()) { dstParent.asDirectory().undoRename4DstParent(removedDst, {code} If the first if check doesn't pass, removedDst would be null and undoRemoveDst may be true. This combination would lead to NullPointerException in the finally block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7522) TestDecommission#testIncludeByRegistrationName sometimes timeout
Ted Yu created HDFS-7522: Summary: TestDecommission#testIncludeByRegistrationName sometimes timeout Key: HDFS-7522 URL: https://issues.apache.org/jira/browse/HDFS-7522 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From >https://builds.apache.org/job/Hadoop-hdfs-trunk/lastCompletedBuild/testReport/org.apache.hadoop.hdfs/TestDecommission/testIncludeByRegistrationName/ > : {code} java.lang.Exception: test timed out after 36 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName(TestDecommission.java:957) {code} In the test output, the following repeated over 300 times: {code} 2014-12-13 18:44:29,910 ERROR datanode.DataNode (BPServiceActor.java:run(836)) - Initialization failed for Block pool BP-408261154-67.195.81.152-1418496249312 (Datanode Uuid null) service to localhost/127.0.0.1:38607 Datanode denied communication with namenode because the host is not in the include-list: DatanodeRegistration(127.0.0.1, datanodeUuid=755318af-3336-462d-9bd7-2a7b966ee4f4, infoPort=45707, infoSecurePort=0, ipcPort=46621, storageInfo=lv=-56;cid=testClusterID;nsid=1154823031;c=0) at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:915) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:4402) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:1196) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:92) at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:26296) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:966) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2127) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2123) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2121) 2014-12-13 18:44:30,871 INFO hdfs.TestDecommission (TestDecommission.java:testIncludeByRegistrationName(956)) - Waiting for datanode to come back 2014-12-13 18:44:31,873 INFO hdfs.TestDecommission (TestDecommission.java:testIncludeByRegistrationName(956)) - Waiting for datanode to come back 2014-12-13 18:44:32,874 INFO hdfs.TestDecommission (TestDecommission.java:testIncludeByRegistrationName(956)) - Waiting for datanode to come back {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7495) Lock inversion in DFSInputStream#getBlockAt()
Ted Yu created HDFS-7495: Summary: Lock inversion in DFSInputStream#getBlockAt() Key: HDFS-7495 URL: https://issues.apache.org/jira/browse/HDFS-7495 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor There're two locks: one on DFSInputStream.this , one on DFSInputStream.infoLock Normally lock is obtained on infoLock, then on DFSInputStream.infoLock However, such order is not observed in DFSInputStream#getBlockAt() : {code} synchronized(infoLock) { ... if (updatePosition) { // synchronized not strictly needed, since we only get here // from synchronized caller methods synchronized(this) { {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7494) Checking of closed in DFSInputStream#pread() should be protected by synchronization
Ted Yu created HDFS-7494: Summary: Checking of closed in DFSInputStream#pread() should be protected by synchronization Key: HDFS-7494 URL: https://issues.apache.org/jira/browse/HDFS-7494 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} private int pread(long position, byte[] buffer, int offset, int length) throws IOException { // sanity checks dfsClient.checkOpen(); if (closed) { {code} Checking of closed should be protected by holding lock on "DFSInputStream.this" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7493) removedDst should be checked against null in finally block of FSDirRenameOp#unprotectedRenameTo()
Ted Yu created HDFS-7493: Summary: removedDst should be checked against null in finally block of FSDirRenameOp#unprotectedRenameTo() Key: HDFS-7493 URL: https://issues.apache.org/jira/browse/HDFS-7493 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} removedDst = dstIIP.getLastINode(); undoRemoveDst = true; {code} If removedDst is null, the following code in finally block may result in NPE: {code} if (dstParent.isDirectory() && dstParent.asDirectory().isWithSnapshot()) { dstParent.asDirectory().undoRename4DstParent(removedDst, dstIIP.getLatestSnapshotId()); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7471) TestDatanodeManager#testNumVersionsReportedCorrect occasionally fails
Ted Yu created HDFS-7471: Summary: TestDatanodeManager#testNumVersionsReportedCorrect occasionally fails Key: HDFS-7471 URL: https://issues.apache.org/jira/browse/HDFS-7471 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1957/ : {code} FAILED: org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager.testNumVersionsReportedCorrect Error Message: The map of version counts returned by DatanodeManager was not what it was expected to be on iteration 237 expected:<0> but was:<1> Stack Trace: java.lang.AssertionError: The map of version counts returned by DatanodeManager was not what it was expected to be on iteration 237 expected:<0> but was:<1> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager.testNumVersionsReportedCorrect(TestDatanodeManager.java:150) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7464) TestDFSAdminWithHA#testRefreshSuperUserGroupsConfiguration fails against Java 8
Ted Yu created HDFS-7464: Summary: TestDFSAdminWithHA#testRefreshSuperUserGroupsConfiguration fails against Java 8 Key: HDFS-7464 URL: https://issues.apache.org/jira/browse/HDFS-7464 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/23/ : {code} REGRESSION: org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA.testRefreshSuperUserGroupsConfiguration Error Message: refreshSuperUserGroupsConfiguration: End of File Exception between local host is: "asf908.gq1.ygridcore.net/67.195.81.152"; destination host is: "localhost":12700; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException expected:<0> but was:<-1> Stack Trace: java.lang.AssertionError: refreshSuperUserGroupsConfiguration: End of File Exception between local host is: "asf908.gq1.ygridcore.net/67.195.81.152"; destination host is: "localhost":12700; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException expected:<0> but was:<-1> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA.testRefreshSuperUserGroupsConfiguration(TestDFSAdminWithHA.java:228) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7455) TestReplaceDatanodeOnFailure sometimes fails
Ted Yu created HDFS-7455: Summary: TestReplaceDatanodeOnFailure sometimes fails Key: HDFS-7455 URL: https://issues.apache.org/jira/browse/HDFS-7455 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1947/ : {code} REGRESSION: org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure.testReplaceDatanodeOnFailure Error Message: expected:<3> but was:<2> Stack Trace: java.lang.AssertionError: expected:<3> but was:<2> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure$SlowWriter.checkReplication(TestReplaceDatanodeOnFailure.java:235) at org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure.testReplaceDatanodeOnFailure(TestReplaceDatanodeOnFailure.java:154) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7448) TestBookKeeperHACheckpoints fails in trunk build
Ted Yu created HDFS-7448: Summary: TestBookKeeperHACheckpoints fails in trunk build Key: HDFS-7448 URL: https://issues.apache.org/jira/browse/HDFS-7448 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor The test failed against both java 7 and java 8. >From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/17/console : {code} testStandbyExceptionThrownDuringCheckpoint(org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints) Time elapsed: 6.822 sec <<< ERROR! org.apache.hadoop.ipc.RemoteException: File /testFile could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1558) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3024) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:699) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:966) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2125) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2121) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2119) at org.apache.hadoop.ipc.Client.call(Client.java:1468) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) at com.sun.proxy.$Proxy20.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101) at com.sun.proxy.$Proxy21.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1544) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:600) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7422) TestEncryptionZonesWithKMS fails against Java 8
Ted Yu created HDFS-7422: Summary: TestEncryptionZonesWithKMS fails against Java 8 Key: HDFS-7422 URL: https://issues.apache.org/jira/browse/HDFS-7422 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/12/ : {code} REGRESSION: org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS.testReadWriteUsingWebHdfs Error Message: Stream closed. Stack Trace: java.io.IOException: Stream closed. at sun.reflect.GeneratedConstructorAccessor58.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:385) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:91) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:656) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:622) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$UnresolvedUrlOpener.connect(WebHdfsFileSystem.java:1204) at org.apache.hadoop.hdfs.web.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:120) at org.apache.hadoop.hdfs.web.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:104) at org.apache.hadoop.hdfs.web.ByteRangeInputStream.(ByteRangeInputStream.java:89) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$OffsetUrlInputStream.(WebHdfsFileSystem.java:1261) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.open(WebHdfsFileSystem.java:1175) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) at org.apache.hadoop.hdfs.DFSTestUtil.verifyFilesEqual(DFSTestUtil.java:1399) at org.apache.hadoop.hdfs.TestEncryptionZones.testReadWriteUsingWebHdfs(TestEncryptionZones.java:634) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Caused by: org.apache.hadoop.ipc.RemoteException: Stream closed. at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:165) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:353) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$UnresolvedUrlOpener.connect(WebHdfsFileSystem.java:1204) at org.apache.hadoop.hdfs.web.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:120) at org.apache.hadoop.hdfs.web.ByteRangeIn
[jira] [Created] (HDFS-7399) Lack of synchronization in DFSOutputStream#Packet#getLastByteOffsetBlock()
Ted Yu created HDFS-7399: Summary: Lack of synchronization in DFSOutputStream#Packet#getLastByteOffsetBlock() Key: HDFS-7399 URL: https://issues.apache.org/jira/browse/HDFS-7399 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} long getLastByteOffsetBlock() { return offsetInBlock + dataPos - dataStart; {code} Access to fields of Packet.this should be protected by synchronization as done in other methods such as writeTo(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7393) TestDFSUpgradeFromImage#testUpgradeFromCorruptRel22Image fails in trunk
Ted Yu created HDFS-7393: Summary: TestDFSUpgradeFromImage#testUpgradeFromCorruptRel22Image fails in trunk Key: HDFS-7393 URL: https://issues.apache.org/jira/browse/HDFS-7393 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor The following is reproducible: {code} Running org.apache.hadoop.hdfs.TestDFSUpgradeFromImage Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.017 sec <<< FAILURE! - in org.apache.hadoop.hdfs.TestDFSUpgradeFromImage testUpgradeFromCorruptRel22Image(org.apache.hadoop.hdfs.TestDFSUpgradeFromImage) Time elapsed: 1.005 sec <<< ERROR! java.lang.IllegalStateException: null at com.google.common.base.Preconditions.checkState(Preconditions.java:129) at org.apache.hadoop.hdfs.server.blockmanagement.BlockIdManager.setGenerationStampV1Limit(BlockIdManager.java:85) at org.apache.hadoop.hdfs.server.blockmanagement.BlockIdManager.clear(BlockIdManager.java:206) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.clear(FSNamesystem.java:622) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:667) at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:376) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:268) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:991) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:537) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:596) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:763) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:747) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1443) at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1104) at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:975) at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:804) at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:465) at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:424) at org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.upgradeAndVerify(TestDFSUpgradeFromImage.java:582) at org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.testUpgradeFromCorruptRel22Image(TestDFSUpgradeFromImage.java:318) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7311) TestLeaseRecovery2 sometimes fails in trunk
Ted Yu created HDFS-7311: Summary: TestLeaseRecovery2 sometimes fails in trunk Key: HDFS-7311 URL: https://issues.apache.org/jira/browse/HDFS-7311 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1917/ : {code} REGRESSION: org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery Error Message: Call From asf909.gq1.ygridcore.net/67.195.81.153 to localhost:55061 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused Stack Trace: java.net.ConnectException: Call From asf909.gq1.ygridcore.net/67.195.81.153 to localhost:55061 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521) at org.apache.hadoop.ipc.Client.call(Client.java:1438) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230) at com.sun.proxy.$Proxy19.create(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:295) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101) at com.sun.proxy.$Proxy20.create(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1694) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1654) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1579) at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:397) at org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:393) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:393) at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:337) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889) at org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery(TestLeaseRecovery2.java:276) FAILED: org.apache.hadoop.hdfs.TestLeaseRecovery2.org.apache.hadoop.hdfs.TestLeaseRecovery2 Error Message: Test resulted in an unexpected exit Stack Trace: java.lang.AssertionError: Test resulted in an unexpected exit at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1709) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1696) at org.apache.hadoop.hdfs.TestLeaseRecovery2.tearDown(TestLeaseRecovery2.java:105) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7289) TestDFSUpgradeWithHA sometimes fails in trunk
Ted Yu created HDFS-7289: Summary: TestDFSUpgradeWithHA sometimes fails in trunk Key: HDFS-7289 URL: https://issues.apache.org/jira/browse/HDFS-7289 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From trunk build #1912: {code} REGRESSION: org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA.testFinalizeFromSecondNameNodeWithJournalNodes Error Message: java.lang.RuntimeException: java.net.SocketTimeoutException: Read timed out Stack Trace: java.io.IOException: java.lang.RuntimeException: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:698) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:641) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1218) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379) at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.doGetUrl(TransferFsImage.java:410) at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:395) at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.downloadImageToStorage(TransferFsImage.java:114) at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.doRun(BootstrapStandby.java:213) at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.access$000(BootstrapStandby.java:69) at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby$1.run(BootstrapStandby.java:107) at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby$1.run(BootstrapStandby.java:103) at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:414) at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:315) at org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA.testFinalizeFromSecondNameNodeWithJournalNodes(TestDFSUpgradeWithHA.java:493) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7267) TestBalancer#testUnknownDatanode occasionally fails in trunk
Ted Yu created HDFS-7267: Summary: TestBalancer#testUnknownDatanode occasionally fails in trunk Key: HDFS-7267 URL: https://issues.apache.org/jira/browse/HDFS-7267 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor In build #1907: {code} REGRESSION: org.apache.hadoop.hdfs.server.balancer.TestBalancer.testUnknownDatanode Error Message: expected:<0> but was:<-3> Stack Trace: java.lang.AssertionError: expected:<0> but was:<-3> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.testUnknownDatanode(TestBalancer.java:737) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7261) storageMap is accessed without synchronization in DatanodeDescriptor#updateHeartbeatState()
Ted Yu created HDFS-7261: Summary: storageMap is accessed without synchronization in DatanodeDescriptor#updateHeartbeatState() Key: HDFS-7261 URL: https://issues.apache.org/jira/browse/HDFS-7261 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is the code: {code} failedStorageInfos = new HashSet( storageMap.values()); {code} In other places, the lock on "DatanodeDescriptor.storageMap" is held: {code} synchronized (storageMap) { final Collection storages = storageMap.values(); return storages.toArray(new DatanodeStorageInfo[storages.size()]); } {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7220) TestDataNodeMetrics fails in trunk
Ted Yu created HDFS-7220: Summary: TestDataNodeMetrics fails in trunk Key: HDFS-7220 URL: https://issues.apache.org/jira/browse/HDFS-7220 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From >https://builds.apache.org/job/Hadoop-Hdfs-trunk/1896/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeMetrics/testSendDataPacketMetrics/ > : {code} java.lang.NoClassDefFoundError: org/apache/hadoop/util/IntrusiveCollection$IntrusiveIterator at org.apache.hadoop.util.IntrusiveCollection.iterator(IntrusiveCollection.java:213) at org.apache.hadoop.util.IntrusiveCollection.clear(IntrusiveCollection.java:368) at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.clearPendingCachingCommands(DatanodeManager.java:1590) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1262) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.close(FSNamesystem.java:1590) at org.apache.hadoop.hdfs.server.namenode.NameNode.stopCommonServices(NameNode.java:658) at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:823) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1717) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1696) at org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics.testSendDataPacketMetrics(TestDataNodeMetrics.java:94) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7103) TestStorageMover sometimes fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-7103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-7103. -- Resolution: Duplicate Dup of HDFS-7095 > TestStorageMover sometimes fails in trunk > - > > Key: HDFS-7103 > URL: https://issues.apache.org/jira/browse/HDFS-7103 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1876/console : > {code} > Running org.apache.hadoop.hdfs.server.mover.TestStorageMover > Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 125.179 sec > <<< FAILURE! - in org.apache.hadoop.hdfs.server.mover.TestStorageMover > testNoSpaceArchive(org.apache.hadoop.hdfs.server.mover.TestStorageMover) > Time elapsed: 11.695 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.hdfs.server.mover.TestStorageMover.testNoSpaceArchive(TestStorageMover.java:720) > testNoSpaceDisk(org.apache.hadoop.hdfs.server.mover.TestStorageMover) Time > elapsed: 11.596 sec <<< FAILURE! > java.lang.AssertionError: null > at org.junit.Assert.fail(Assert.java:86) > at org.junit.Assert.assertTrue(Assert.java:41) > at org.junit.Assert.assertTrue(Assert.java:52) > at > org.apache.hadoop.hdfs.server.mover.TestStorageMover.testNoSpaceDisk(TestStorageMover.java:637) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7103) TestStorageMover sometimes fails in trunk
Ted Yu created HDFS-7103: Summary: TestStorageMover sometimes fails in trunk Key: HDFS-7103 URL: https://issues.apache.org/jira/browse/HDFS-7103 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1876/console : {code} Running org.apache.hadoop.hdfs.server.mover.TestStorageMover Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 125.179 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.mover.TestStorageMover testNoSpaceArchive(org.apache.hadoop.hdfs.server.mover.TestStorageMover) Time elapsed: 11.695 sec <<< FAILURE! java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.hadoop.hdfs.server.mover.TestStorageMover.testNoSpaceArchive(TestStorageMover.java:720) testNoSpaceDisk(org.apache.hadoop.hdfs.server.mover.TestStorageMover) Time elapsed: 11.596 sec <<< FAILURE! java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.hadoop.hdfs.server.mover.TestStorageMover.testNoSpaceDisk(TestStorageMover.java:637) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7102) Null dereference in PacketReceiver#receiveNextPacket()
Ted Yu created HDFS-7102: Summary: Null dereference in PacketReceiver#receiveNextPacket() Key: HDFS-7102 URL: https://issues.apache.org/jira/browse/HDFS-7102 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} public void receiveNextPacket(ReadableByteChannel in) throws IOException { doRead(in, null); {code} doRead() would pass null as second parameter to (line 134): {code} doReadFully(ch, in, curPacketBuf); {code} which dereferences it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7101) Potential null dereference in DFSck#doWork()
Ted Yu created HDFS-7101: Summary: Potential null dereference in DFSck#doWork() Key: HDFS-7101 URL: https://issues.apache.org/jira/browse/HDFS-7101 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} String lastLine = null; int errCode = -1; try { while ((line = input.readLine()) != null) { ... if (lastLine.endsWith(NamenodeFsck.HEALTHY_STATUS)) { errCode = 0; {code} If readLine() throws exception, lastLine may be null, leading to NPE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7083) TestDecommission#testIncludeByRegistrationName sometimes fails
Ted Yu created HDFS-7083: Summary: TestDecommission#testIncludeByRegistrationName sometimes fails Key: HDFS-7083 URL: https://issues.apache.org/jira/browse/HDFS-7083 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1874/ : {code} REGRESSION: org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName Error Message: test timed out after 36 milliseconds Stack Trace: java.lang.Exception: test timed out after 36 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName(TestDecommission.java:957) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7008) xlator should be closed upon exit from DFSAdmin#genericRefresh()
Ted Yu created HDFS-7008: Summary: xlator should be closed upon exit from DFSAdmin#genericRefresh() Key: HDFS-7008 URL: https://issues.apache.org/jira/browse/HDFS-7008 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} GenericRefreshProtocol xlator = new GenericRefreshProtocolClientSideTranslatorPB(proxy); // Refresh Collection responses = xlator.refresh(identifier, args); {code} GenericRefreshProtocolClientSideTranslatorPB#close() should be called on xlator before return. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-6902) FileWriter should be closed in finally block in BlockReceiver#receiveBlock()
Ted Yu created HDFS-6902: Summary: FileWriter should be closed in finally block in BlockReceiver#receiveBlock() Key: HDFS-6902 URL: https://issues.apache.org/jira/browse/HDFS-6902 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is code starting from line 828: {code} try { FileWriter out = new FileWriter(restartMeta); // write out the current time. out.write(Long.toString(Time.now() + restartBudget)); out.flush(); out.close(); } catch (IOException ioe) { {code} If write() or flush() call throws IOException, out wouldn't be closed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6852) TestBalancerWithEncryptedTransfer sometimes fails in trunk
Ted Yu created HDFS-6852: Summary: TestBalancerWithEncryptedTransfer sometimes fails in trunk Key: HDFS-6852 URL: https://issues.apache.org/jira/browse/HDFS-6852 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1837/ : {code} REGRESSION: org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer.testEncryptedBalancer0 Error Message: Rebalancing expected avg utilization to become 0.2, but on datanode 127.0.0.1:47596 it remains at 0.08 after more than 4 msec. Stack Trace: java.util.concurrent.TimeoutException: Rebalancing expected avg utilization to become 0.2, but on datanode 127.0.0.1:47596 it remains at 0.08 after more than 4 msec. at org.apache.hadoop.hdfs.server.balancer.TestBalancer.waitForBalancer(TestBalancer.java:321) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.runBalancer(TestBalancer.java:580) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:551) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:437) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.oneNodeTest(TestBalancer.java:645) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancer0Internal(TestBalancer.java:759) at org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer.testEncryptedBalancer0(TestBalancerWithEncryptedTransfer.java:38) {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6848) Lack of synchronization on access to datanodeUuid in DataStorage#format()
Ted Yu created HDFS-6848: Summary: Lack of synchronization on access to datanodeUuid in DataStorage#format() Key: HDFS-6848 URL: https://issues.apache.org/jira/browse/HDFS-6848 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} this.datanodeUuid = datanodeUuid; {code} The above assignment should be done holding lock "DataStorage.this" - as is done in two other places. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-6842) TestHttpFSFWithWebhdfsFileSystem fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-6842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-6842. -- Resolution: Duplicate Covered by HADOOP-10836 > TestHttpFSFWithWebhdfsFileSystem fails in trunk > --- > > Key: HDFS-6842 > URL: https://issues.apache.org/jira/browse/HDFS-6842 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu > > This can be reproduced locally: > {code} > testOperationDoAs[21](org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem) > Time elapsed: 0.315 sec <<< ERROR! > org.apache.hadoop.ipc.RemoteException: User: zy is not allowed to impersonate > user1 > at org.apache.hadoop.ipc.Client.call(Client.java:1411) > at org.apache.hadoop.ipc.Client.call(Client.java:1364) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy24.mkdirs(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:512) > at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101) > at com.sun.proxy.$Proxy25.mkdirs(Unknown Source) > at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2546) > at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2517) > at > org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:821) > at > org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:817) > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6842) TestHttpFSFWithWebhdfsFileSystem fails in trunk
Ted Yu created HDFS-6842: Summary: TestHttpFSFWithWebhdfsFileSystem fails in trunk Key: HDFS-6842 URL: https://issues.apache.org/jira/browse/HDFS-6842 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu This can be reproduced locally: {code} testOperationDoAs[21](org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem) Time elapsed: 0.315 sec <<< ERROR! org.apache.hadoop.ipc.RemoteException: User: zy is not allowed to impersonate user1 at org.apache.hadoop.ipc.Client.call(Client.java:1411) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy24.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:512) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101) at com.sun.proxy.$Proxy25.mkdirs(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2546) at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2517) at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:821) at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:817) {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6810) storageMap is accessed without proper synchronization in DatanodeDescriptor#getStorageReports
Ted Yu created HDFS-6810: Summary: storageMap is accessed without proper synchronization in DatanodeDescriptor#getStorageReports Key: HDFS-6810 URL: https://issues.apache.org/jira/browse/HDFS-6810 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor Here is related code: {code} public StorageReport[] getStorageReports() { final StorageReport[] reports = new StorageReport[storageMap.size()]; {code} Other methods use the following construct: {code} synchronized (storageMap) { {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6726) TestNamenodeCapacityReport fails intermittently
Ted Yu created HDFS-6726: Summary: TestNamenodeCapacityReport fails intermittently Key: HDFS-6726 URL: https://issues.apache.org/jira/browse/HDFS-6726 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From >https://builds.apache.org/job/Hadoop-Hdfs-trunk/1812/testReport/junit/org.apache.hadoop.hdfs.server.namenode/TestNamenodeCapacityReport/testXceiverCount/ > : {code} java.io.IOException: Unable to close file because the last block does not have enough number of replicas. at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2141) at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2109) at org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport.testXceiverCount(TestNamenodeCapacityReport.java:281) {code} There were multiple occurrences of 'Broken pipe', 'Connection reset by peer' and 'Premature EOF from inputStream' exceptions in test output -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6586) TestBalancer#testExitZeroOnSuccess sometimes fails in trunk
Ted Yu created HDFS-6586: Summary: TestBalancer#testExitZeroOnSuccess sometimes fails in trunk Key: HDFS-6586 URL: https://issues.apache.org/jira/browse/HDFS-6586 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From >https://builds.apache.org/job/Hadoop-Hdfs-trunk/1782/testReport/org.apache.hadoop.hdfs.server.balancer/TestBalancer/testExitZeroOnSuccess/ > : {code} Stacktrace java.util.concurrent.TimeoutException: Rebalancing expected avg utilization to become 0.2, but on datanode 127.0.0.1:49048 it remains at 0.08 after more than 4 msec. at org.apache.hadoop.hdfs.server.balancer.TestBalancer.waitForBalancer(TestBalancer.java:284) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.runBalancerCli(TestBalancer.java:392) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:357) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.oneNodeTest(TestBalancer.java:398) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.testExitZeroOnSuccess(TestBalancer.java:550) {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6582) Missing null check in RpcProgramNfs3#read(XDR, SecurityHandler)
Ted Yu created HDFS-6582: Summary: Missing null check in RpcProgramNfs3#read(XDR, SecurityHandler) Key: HDFS-6582 URL: https://issues.apache.org/jira/browse/HDFS-6582 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor Around line 691: {code} FSDataInputStream fis = clientCache.getDfsInputStream(userName, Nfs3Utils.getFileIdPath(handle)); try { readCount = fis.read(offset, readbuffer, 0, count); {code} fis may be null, leading to NullPointerException -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6501) TestCrcCorruption#testCorruptionDuringWrt sometimes fails in trunk
Ted Yu created HDFS-6501: Summary: TestCrcCorruption#testCorruptionDuringWrt sometimes fails in trunk Key: HDFS-6501 URL: https://issues.apache.org/jira/browse/HDFS-6501 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1767/ : {code} REGRESSION: org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt Error Message: test timed out after 5 milliseconds Stack Trace: java.lang.Exception: test timed out after 5 milliseconds at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2024) at org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2008) at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2107) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:98) at org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt(TestCrcCorruption.java:133) {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6481) DatanodeManager#getDatanodeStorageInfos() should check the length of storageIDs
Ted Yu created HDFS-6481: Summary: DatanodeManager#getDatanodeStorageInfos() should check the length of storageIDs Key: HDFS-6481 URL: https://issues.apache.org/jira/browse/HDFS-6481 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.3.0 Reporter: Ted Yu Ian Brooks reported the following stack trace: {code} 2014-06-03 13:05:03,915 WARN [DataStreamer for file /user/hbase/WALs/,16020,1401716790638/%2C16020%2C1401716790638.1401796562200 block BP-2121456822-10.143.38.149-1396953188241:blk_1074073683_332932] hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.lang.ArrayIndexOutOfBoundsException): 0 at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getDatanodeStorageInfos(DatanodeManager.java:467) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2779) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:594) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:430) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1958) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1956) at org.apache.hadoop.ipc.Client.call(Client.java:1347) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy13.getAdditionalDatanode(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolTranslatorPB.java:352) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy14.getAdditionalDatanode(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:266) at com.sun.proxy.$Proxy15.getAdditionalDatanode(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:919) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:919) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1031) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:823) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:475) 2014-06-03 13:05:48,489 ERROR [RpcServer.handler=22,port=16020] wal.FSHLog: syncer encountered error, will retry. txid=211 org.apache.hadoop.ipc.RemoteException(java.lang.ArrayIndexOutOfBoundsException): 0 at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getDatanodeStorageInfos(DatanodeManager.java:467) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2779) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:594) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:430) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.cal
[jira] [Created] (HDFS-6437) TestBookKeeperHACheckpoints#TestStandbyCheckpoints fails in trunk
Ted Yu created HDFS-6437: Summary: TestBookKeeperHACheckpoints#TestStandbyCheckpoints fails in trunk Key: HDFS-6437 URL: https://issues.apache.org/jira/browse/HDFS-6437 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor The following test failure can be reproduced locally: {code} testSBNCheckpoints(org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints) Time elapsed: 2.79 sec <<< ERROR! java.lang.NullPointerException: null at org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints.testSBNCheckpoints(TestStandbyCheckpoints.java:138) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6415) Missing null check in FSImageSerialization#writePermissionStatus()
Ted Yu created HDFS-6415: Summary: Missing null check in FSImageSerialization#writePermissionStatus() Key: HDFS-6415 URL: https://issues.apache.org/jira/browse/HDFS-6415 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} PermissionStatus.write(out, inode.getUserName(), inode.getGroupName(), p); {code} getUserName() / getGroupName() may return null. null check should be added for these two calls. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6368) TransferFsImage#receiveFile() should perform validation on fsImageName parameter
Ted Yu created HDFS-6368: Summary: TransferFsImage#receiveFile() should perform validation on fsImageName parameter Key: HDFS-6368 URL: https://issues.apache.org/jira/browse/HDFS-6368 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor Currently only null check is performed: {code} if (fsImageName == null) { throw new IOException("No filename header provided by server"); } newLocalPaths.add(new File(localPath, fsImageName)); {code} Value of fsImageName, obtained from HttpURLConnection header, may be tainted. This may allow an attacker to access, modify, or test the existence of critical or sensitive files. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-6083) TestQuorumJournalManager#testChangeWritersLogsOutOfSync2 occasionally fails
[ https://issues.apache.org/jira/browse/HDFS-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-6083. -- Resolution: Cannot Reproduce > TestQuorumJournalManager#testChangeWritersLogsOutOfSync2 occasionally fails > --- > > Key: HDFS-6083 > URL: https://issues.apache.org/jira/browse/HDFS-6083 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > > From > https://builds.apache.org/job/Hadoop-Hdfs-trunk/1695/testReport/junit/org.apache.hadoop.hdfs.qjournal.client/TestQuorumJournalManager/testChangeWritersLogsOutOfSync2/ > : > {code} > Leaked thread: "IPC Client (26533782) connection to /127.0.0.1:57898 from > jenkins" Id=590 RUNNABLE > at java.lang.System.arraycopy(Native Method) > at java.lang.ThreadGroup.remove(ThreadGroup.java:885) > at java.lang.Thread.exit(Thread.java:672) > {code} > The following check should give more time for the threads to shutdown: > {code} > // Should not leak clients between tests -- this can cause flaky tests. > // (See HDFS-4643) > GenericTestUtils.assertNoThreadsMatching(".*IPC Client.*"); > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6369) RemoteBlockReader#available() should call FSInputChecker.available()
Ted Yu created HDFS-6369: Summary: RemoteBlockReader#available() should call FSInputChecker.available() Key: HDFS-6369 URL: https://issues.apache.org/jira/browse/HDFS-6369 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Trivial Currently DFSClient.TCP_WINDOW_SIZE is directly returned. However, FSInputChecker.available(), in the superclass, may return value lower than the constant. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6291) FSImage may be left unclosed in BootstrapStandby#doRun()
Ted Yu created HDFS-6291: Summary: FSImage may be left unclosed in BootstrapStandby#doRun() Key: HDFS-6291 URL: https://issues.apache.org/jira/browse/HDFS-6291 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor At around line 203: {code} if (!checkLogsAvailableForRead(image, imageTxId, curTxId)) { return ERR_CODE_LOGS_UNAVAILABLE; } {code} If we return following the above check, image is not closed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6290) File is not closed in OfflineImageViewerPB#run()
Ted Yu created HDFS-6290: Summary: File is not closed in OfflineImageViewerPB#run() Key: HDFS-6290 URL: https://issues.apache.org/jira/browse/HDFS-6290 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} } else if (processor.equals("XML")) { new PBImageXmlWriter(conf, out).visit(new RandomAccessFile(inputFile, "r")); {code} The RandomAccessFile instance should be closed before the method returns. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6264) Provide FileSystem#create() variant which throws exception if parent directory doesn't exist
Ted Yu created HDFS-6264: Summary: Provide FileSystem#create() variant which throws exception if parent directory doesn't exist Key: HDFS-6264 URL: https://issues.apache.org/jira/browse/HDFS-6264 Project: Hadoop HDFS Issue Type: Task Reporter: Ted Yu Priority: Minor FileSystem#createNonRecursive() is deprecated. However, there is no DistributedFileSystem#create() implementation which throws exception if parent directory doesn't exist. This limits clients' migration away from the deprecated method. Variant of create() method should be added which throws exception if parent directory doesn't exist. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6257) TestCacheDirectives#testExceedsCapacity fails occasionally in trunk
Ted Yu created HDFS-6257: Summary: TestCacheDirectives#testExceedsCapacity fails occasionally in trunk Key: HDFS-6257 URL: https://issues.apache.org/jira/browse/HDFS-6257 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1736/ : REGRESSION: org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.testExceedsCapacity {code} Error Message: Namenode should not send extra CACHE commands expected:<0> but was:<2> Stack Trace: java.lang.AssertionError: Namenode should not send extra CACHE commands expected:<0> but was:<2> at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:128) at org.junit.Assert.assertEquals(Assert.java:472) at org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.testExceedsCapacity(TestCacheDirectives.java:1419) {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6177) TestHttpFSServer fails occasionally in trunk
Ted Yu created HDFS-6177: Summary: TestHttpFSServer fails occasionally in trunk Key: HDFS-6177 URL: https://issues.apache.org/jira/browse/HDFS-6177 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From https://builds.apache.org/job/Hadoop-hdfs-trunk/1716/consoleFull : {code} Running org.apache.hadoop.fs.http.server.TestHttpFSServer Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.424 sec <<< FAILURE! - in org.apache.hadoop.fs.http.server.TestHttpFSServer testDelegationTokenOperations(org.apache.hadoop.fs.http.server.TestHttpFSServer) Time elapsed: 0.559 sec <<< FAILURE! java.lang.AssertionError: expected:<401> but was:<403> at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:128) at org.junit.Assert.assertEquals(Assert.java:472) at org.junit.Assert.assertEquals(Assert.java:456) at org.apache.hadoop.fs.http.server.TestHttpFSServer.testDelegationTokenOperations(TestHttpFSServer.java:352) {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6160) TestSafeMode occasionally fails
Ted Yu created HDFS-6160: Summary: TestSafeMode occasionally fails Key: HDFS-6160 URL: https://issues.apache.org/jira/browse/HDFS-6160 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From >https://builds.apache.org/job/PreCommit-HDFS-Build/6511//testReport/org.apache.hadoop.hdfs/TestSafeMode/testInitializeReplQueuesEarly/ > : {code} java.lang.AssertionError: expected:<13> but was:<0> at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:128) at org.junit.Assert.assertEquals(Assert.java:472) at org.junit.Assert.assertEquals(Assert.java:456) at org.apache.hadoop.hdfs.TestSafeMode.testInitializeReplQueuesEarly(TestSafeMode.java:212) {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (HDFS-5672) TestHASafeMode#testSafeBlockTracking fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HDFS-5672: -- > TestHASafeMode#testSafeBlockTracking fails in trunk > --- > > Key: HDFS-5672 > URL: https://issues.apache.org/jira/browse/HDFS-5672 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu > > From build #1614: > {code} > TestHASafeMode.testSafeBlockTracking:623->assertSafeMode:488 Bad safemode > status: 'Safe mode is ON. The reported blocks 3 needs additional 7 blocks to > reach the threshold 0.9990 of total blocks 10. > Safe mode will be turned off automatically' > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6092) DistributedFileSystem#getCanonicalServiceName() and DistributedFileSystem#getUri() may return inconsistent results w.r.t. port
Ted Yu created HDFS-6092: Summary: DistributedFileSystem#getCanonicalServiceName() and DistributedFileSystem#getUri() may return inconsistent results w.r.t. port Key: HDFS-6092 URL: https://issues.apache.org/jira/browse/HDFS-6092 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.3.0 Reporter: Ted Yu I discovered this when working on HBASE-10717 Here is sample code to reproduce the problem: {code} Path desPath = new Path("hdfs://127.0.0.1/"); FileSystem desFs = desPath.getFileSystem(conf); String s = desFs.getCanonicalServiceName(); URI uri = desFs.getUri(); {code} Canonical name string contains the default port - 8020 But uri doesn't contain port. This would result in the following exception: {code} testIsSameHdfs(org.apache.hadoop.hbase.util.TestFSHDFSUtils) Time elapsed: 0.001 sec <<< ERROR! java.lang.IllegalArgumentException: port out of range:-1 at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143) at java.net.InetSocketAddress.(InetSocketAddress.java:224) at org.apache.hadoop.hbase.util.FSHDFSUtils.getNNAddresses(FSHDFSUtils.java:88) {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6083) TestQuorumJournalManager#testChangeWritersLogsOutOfSync2 occasionally fails
Ted Yu created HDFS-6083: Summary: TestQuorumJournalManager#testChangeWritersLogsOutOfSync2 occasionally fails Key: HDFS-6083 URL: https://issues.apache.org/jira/browse/HDFS-6083 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From >https://builds.apache.org/job/Hadoop-Hdfs-trunk/1695/testReport/junit/org.apache.hadoop.hdfs.qjournal.client/TestQuorumJournalManager/testChangeWritersLogsOutOfSync2/ > : {code} Leaked thread: "IPC Client (26533782) connection to /127.0.0.1:57898 from jenkins" Id=590 RUNNABLE at java.lang.System.arraycopy(Native Method) at java.lang.ThreadGroup.remove(ThreadGroup.java:885) at java.lang.Thread.exit(Thread.java:672) {code} The following check should give more time for the threads to shutdown: {code} // Should not leak clients between tests -- this can cause flaky tests. // (See HDFS-4643) GenericTestUtils.assertNoThreadsMatching(".*IPC Client.*"); {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6081) TestRetryCacheWithHA#testCreateSymlink occasionally fails in trunk
Ted Yu created HDFS-6081: Summary: TestRetryCacheWithHA#testCreateSymlink occasionally fails in trunk Key: HDFS-6081 URL: https://issues.apache.org/jira/browse/HDFS-6081 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu >From >https://builds.apache.org/job/Hadoop-Hdfs-trunk/1696/testReport/junit/org.apache.hadoop.hdfs.server.namenode.ha/TestRetryCacheWithHA/testCreateSymlink/ > : {code} 2014-03-09 13:18:47,515 WARN security.UserGroupInformation (UserGroupInformation.java:doAs(1600)) - PriviledgedActionException as:jenkins (auth:SIMPLE) cause:java.io.IOException: failed to create link /testlink either because the filename is invalid or the file exists 2014-03-09 13:18:47,515 INFO ipc.Server (Server.java:run(2093)) - IPC Server handler 0 on 39303, call org.apache.hadoop.hdfs.protocol.ClientProtocol.createSymlink from 127.0.0.1:32909 Call#682 Retry#1: error: java.io.IOException: failed to create link /testlink either because the filename is invalid or the file exists java.io.IOException: failed to create link /testlink either because the filename is invalid or the file exists at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlinkInt(FSNamesystem.java:2053) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlink(FSNamesystem.java:2023) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.createSymlink(NameNodeRpcServer.java:965) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.createSymlink(ClientNamenodeProtocolServerSideTranslatorPB.java:844) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:605) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2071) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2067) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1597) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2065) 2014-03-09 13:18:47,522 INFO blockmanagement.BlockManager (BlockManager.java:processMisReplicatesAsync(2475)) - Total number of blocks = 1 2014-03-09 13:18:47,523 INFO blockmanagement.BlockManager (BlockManager.java:processMisReplicatesAsync(2476)) - Number of invalid blocks = 0 2014-03-09 13:18:47,523 INFO blockmanagement.BlockManager (BlockManager.java:processMisReplicatesAsync(2477)) - Number of under-replicated blocks = 0 2014-03-09 13:18:47,523 INFO ha.TestRetryCacheWithHA (TestRetryCacheWithHA.java:run(1162)) - Got Exception while calling createSymlink org.apache.hadoop.ipc.RemoteException(java.io.IOException): failed to create link /testlink either because the filename is invalid or the file exists at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlinkInt(FSNamesystem.java:2053) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlink(FSNamesystem.java:2023) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.createSymlink(NameNodeRpcServer.java:965) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.createSymlink(ClientNamenodeProtocolServerSideTranslatorPB.java:844) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:605) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2071) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2067) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1597) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2065) at org.apache.hadoop.ipc.Client.call(Client.java:1409) at org.apache.hadoop.ipc.Client.call(Client.java:1362) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at $Proxy17.createSymlink(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.createSymlink(ClientNamenodeProtocolTranslatorPB.java:794) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAcces
[jira] [Created] (HDFS-6037) TestIncrementalBlockReports#testReplaceReceivedBlock fails occasionally in trunk
Ted Yu created HDFS-6037: Summary: TestIncrementalBlockReports#testReplaceReceivedBlock fails occasionally in trunk Key: HDFS-6037 URL: https://issues.apache.org/jira/browse/HDFS-6037 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu >From >https://builds.apache.org/job/Hadoop-Hdfs-trunk/1688/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestIncrementalBlockReports/testReplaceReceivedBlock/ > : {code} datanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted( , , ); Wanted 1 time: -> at org.apache.hadoop.hdfs.server.datanode.TestIncrementalBlockReports.testReplaceReceivedBlock(TestIncrementalBlockReports.java:198) But was 2 times. Undesired invocation: -> at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.reportReceivedDeletedBlocks(BPServiceActor.java:303) {code} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5997) TestHASafeMode#testBlocksAddedWhileStandbyIsDown fails in trunk
Ted Yu created HDFS-5997: Summary: TestHASafeMode#testBlocksAddedWhileStandbyIsDown fails in trunk Key: HDFS-5997 URL: https://issues.apache.org/jira/browse/HDFS-5997 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu >From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1681/ : REGRESSION: org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode.testBlocksAddedWhileStandbyIsDown Error Message: {code} Bad safemode status: 'Safe mode is ON. The reported blocks 7 has reached the threshold 0.9990 of total blocks 6. The number of live datanodes 3 has reached the minimum number 0. Safe mode will be turned off automatically in 28 seconds.' {code} Stack Trace: {code} java.lang.AssertionError: Bad safemode status: 'Safe mode is ON. The reported blocks 7 has reached the threshold 0.9990 of total blocks 6. The number of live datanodes 3 has reached the minimum number 0. Safe mode will be turned off automatically in 28 seconds.' at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode.assertSafeMode(TestHASafeMode.java:493) at org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode.testBlocksAddedWhileStandbyIsDown(TestHASafeMode.java:660) {code} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5953) TestBlockReaderFactory fails in trunk
Ted Yu created HDFS-5953: Summary: TestBlockReaderFactory fails in trunk Key: HDFS-5953 URL: https://issues.apache.org/jira/browse/HDFS-5953 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu >From >https://builds.apache.org/job/Hadoop-Hdfs-trunk/1673/testReport/junit/org.apache.hadoop.hdfs/TestBlockReaderFactory/testFallbackFromShortCircuitToUnixDomainTraffic/ > : {code} java.lang.RuntimeException: Although a UNIX domain socket path is configured as /tmp/socks.1392383436573.1418778351/testFallbackFromShortCircuitToUnixDomainTraffic._PORT, we cannot start a localDataXceiverServer because libhadoop cannot be loaded. at org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:601) at org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:573) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:769) at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:315) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1864) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1764) at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1243) at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:699) at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:359) at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:340) at org.apache.hadoop.hdfs.TestBlockReaderFactory.testFallbackFromShortCircuitToUnixDomainTraffic(TestBlockReaderFactory.java:99) {code} This test failure can be reproduced locally (on Mac). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5913) Nfs3Utils#getWccAttr() should check attr parameter against null
Ted Yu created HDFS-5913: Summary: Nfs3Utils#getWccAttr() should check attr parameter against null Key: HDFS-5913 URL: https://issues.apache.org/jira/browse/HDFS-5913 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor In RpcProgramNfs3#commit() : {code} Nfs3FileAttributes postOpAttr = null; try { postOpAttr = writeManager.getFileAttr(dfsClient, handle, iug); } catch (IOException e1) { LOG.info("Can't get postOpAttr for fileId: " + handle.getFileId()); } WccData fileWcc = new WccData(Nfs3Utils.getWccAttr(preOpAttr), postOpAttr); {code} If there is exception, postOpAttr would be null. However, Nfs3Utils#getWccAttr() dereferences attr parameter directly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5897) TestNNWithQJM#testNewNamenodeTakesOverWriter occasionally fails in trunk
Ted Yu created HDFS-5897: Summary: TestNNWithQJM#testNewNamenodeTakesOverWriter occasionally fails in trunk Key: HDFS-5897 URL: https://issues.apache.org/jira/browse/HDFS-5897 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu >From >https://builds.apache.org/job/Hadoop-Hdfs-trunk/1665/testReport/junit/org.apache.hadoop.hdfs.qjournal/TestNNWithQJM/testNewNamenodeTakesOverWriter/ > : {code} java.lang.Exception: test timed out after 3 milliseconds at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:412) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:401) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) {code} I saw: {code} 2014-02-06 11:38:37,970 ERROR namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(221)) - Got error reading edit log input stream http://localhost:40509/getJournal?jid=myjournal&segmentTxId=3&storageInfo=-51%3A1571339494%3A0%3AtestClusterID; failing over to edit log http://localhost:56244/getJournal?jid=myjournal&segmentTxId=3&storageInfo=-51%3A1571339494%3A0%3AtestClusterID org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 4 at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:83) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:140) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:83) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:167) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:120) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:708) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:606) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:263) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:874) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:634) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:446) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:502) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:658) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:643) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1291) at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:939) at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:824) at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:678) at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:359) at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:340) at org.apache.hadoop.hdfs.qjournal.TestNNWithQJM.testNewNamenodeTakesOverWriter(TestNNWithQJM.java:145) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) {code} -- This me
[jira] [Created] (HDFS-5892) TestDeleteBlockPool fails in branch-2
Ted Yu created HDFS-5892: Summary: TestDeleteBlockPool fails in branch-2 Key: HDFS-5892 URL: https://issues.apache.org/jira/browse/HDFS-5892 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor Running test suite on Linux, I got: {code} testDeleteBlockPool(org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool) Time elapsed: 8.143 sec <<< ERROR! java.io.IOException: All datanodes 127.0.0.1:43721 are bad. Aborting... at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1023) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:838) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:483) {code} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5886) Potential null pointer deference in RpcProgramNfs3#readlink()
Ted Yu created HDFS-5886: Summary: Potential null pointer deference in RpcProgramNfs3#readlink() Key: HDFS-5886 URL: https://issues.apache.org/jira/browse/HDFS-5886 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Here is related code: {code} if (MAX_READ_TRANSFER_SIZE < target.getBytes().length) { return new READLINK3Response(Nfs3Status.NFS3ERR_IO, postOpAttr, null); } {code} READLINK3Response constructor would dereference the third parameter: {code} this.path = new byte[path.length]; {code} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5839) TestWebHDFS#testNamenodeRestart fails with NullPointerException in trunk
Ted Yu created HDFS-5839: Summary: TestWebHDFS#testNamenodeRestart fails with NullPointerException in trunk Key: HDFS-5839 URL: https://issues.apache.org/jira/browse/HDFS-5839 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5834) TestCheckpoint#testCheckpoint may fail due to Bad value assertion
Ted Yu created HDFS-5834: Summary: TestCheckpoint#testCheckpoint may fail due to Bad value assertion Key: HDFS-5834 URL: https://issues.apache.org/jira/browse/HDFS-5834 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor I saw the following when running test suite on Linux: {code} testCheckpoint(org.apache.hadoop.hdfs.server.namenode.TestCheckpoint) Time elapsed: 3.058 sec <<< FAILURE! java.lang.AssertionError: Bad value for metric GetImageNumOps Expected: gt(0) got: <0L> at org.junit.Assert.assertThat(Assert.java:780) at org.apache.hadoop.test.MetricsAsserts.assertCounterGt(MetricsAsserts.java:318) at org.apache.hadoop.hdfs.server.namenode.TestCheckpoint.testCheckpoint(TestCheckpoint.java:1058) {code} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5750) JHLogAnalyzer#parseLogFile() should close stm upon return
Ted Yu created HDFS-5750: Summary: JHLogAnalyzer#parseLogFile() should close stm upon return Key: HDFS-5750 URL: https://issues.apache.org/jira/browse/HDFS-5750 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor stm is assigned to in But stm may point to another InputStream : {code} if(compressionClass != null) { CompressionCodec codec = (CompressionCodec) ReflectionUtils.newInstance(compressionClass, new Configuration()); in = codec.createInputStream(stm); {code} stm should be closed in the finally block. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5721) sharedEditsImage in Namenode#initializeSharedEdits() should be closed before method returns
Ted Yu created HDFS-5721: Summary: sharedEditsImage in Namenode#initializeSharedEdits() should be closed before method returns Key: HDFS-5721 URL: https://issues.apache.org/jira/browse/HDFS-5721 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor At line 901: {code} FSImage sharedEditsImage = new FSImage(conf, Lists.newArrayList(), sharedEditsDirs); {code} sharedEditsImage is not closed before the method returns. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5719) FSImage#doRollback() should close prevState before return
Ted Yu created HDFS-5719: Summary: FSImage#doRollback() should close prevState before return Key: HDFS-5719 URL: https://issues.apache.org/jira/browse/HDFS-5719 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} FSImage prevState = new FSImage(conf); {code} prevState should be closed before return from doRollback() -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5718) TestHttpsFileSystem intermittently fails with Port in use error
Ted Yu created HDFS-5718: Summary: TestHttpsFileSystem intermittently fails with Port in use error Key: HDFS-5718 URL: https://issues.apache.org/jira/browse/HDFS-5718 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu Priority: Minor >From >https://builds.apache.org/job/Hadoop-Hdfs-trunk/1634/testReport/junit/org.apache.hadoop.hdfs.web/TestHttpsFileSystem/org_apache_hadoop_hdfs_web_TestHttpsFileSystem/ > : {code} java.net.BindException: Port in use: localhost:50475 at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383) at java.net.ServerSocket.bind(ServerSocket.java:328) at java.net.ServerSocket.(ServerSocket.java:194) at javax.net.ssl.SSLServerSocket.(SSLServerSocket.java:106) at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.(SSLServerSocketImpl.java:108) at com.sun.net.ssl.internal.ssl.SSLServerSocketFactoryImpl.createServerSocket(SSLServerSocketFactoryImpl.java:72) at org.mortbay.jetty.security.SslSocketConnector.newServerSocket(SslSocketConnector.java:478) at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73) at org.apache.hadoop.http.HttpServer.openListeners(HttpServer.java:973) at org.apache.hadoop.http.HttpServer.start(HttpServer.java:914) at org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:412) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:769) at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:315) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1846) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1746) at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1203) at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:673) at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:342) at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:323) at org.apache.hadoop.hdfs.web.TestHttpsFileSystem.setUp(TestHttpsFileSystem.java:64) {code} This could have been caused by concurrent test(s). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5710) FSDirectory#getFullPathName should check for inodes against null
Ted Yu created HDFS-5710: Summary: FSDirectory#getFullPathName should check for inodes against null Key: HDFS-5710 URL: https://issues.apache.org/jira/browse/HDFS-5710 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.2.0 Reporter: Ted Yu >From >https://builds.apache.org/job/hbase-0.96-hadoop2/166/testReport/junit/org.apache.hadoop.hbase.mapreduce/TestTableInputFormatScan1/org_apache_hadoop_hbase_mapreduce_TestTableInputFormatScan1/ > : {code} 2014-01-01 00:10:15,571 INFO [IPC Server handler 2 on 50198] blockmanagement.BlockManager(1009): BLOCK* addToInvalidates: blk_1073741967_1143 127.0.0.1:40188 127.0.0.1:46149 127.0.0.1:41496 2014-01-01 00:10:16,559 WARN [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@93935b] namenode.FSDirectory(1854): Could not get full path. Corresponding file might have deleted already. 2014-01-01 00:10:16,560 FATAL [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@93935b] blockmanagement.BlockManager$ReplicationMonitor(3127): ReplicationMonitor thread received Runtime exception. java.lang.NullPointerException at org.apache.hadoop.hdfs.server.namenode.FSDirectory.getFullPathName(FSDirectory.java:1871) at org.apache.hadoop.hdfs.server.namenode.INode.getFullPathName(INode.java:482) at org.apache.hadoop.hdfs.server.namenode.INodeFile.getName(INodeFile.java:316) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.chooseTarget(BlockPlacementPolicy.java:118) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWorkForBlocks(BlockManager.java:1259) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWork(BlockManager.java:1167) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3158) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3112) at java.lang.Thread.run(Thread.java:724) {code} Looks like getRelativePathINodes() returned null but getFullPathName() didn't check inodes against null, leading to NPE. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5705) TestSecondaryNameNodeUpgrade#testChangeNsIDFails may fail due to ConcurrentModificationException
Ted Yu created HDFS-5705: Summary: TestSecondaryNameNodeUpgrade#testChangeNsIDFails may fail due to ConcurrentModificationException Key: HDFS-5705 URL: https://issues.apache.org/jira/browse/HDFS-5705 Project: Hadoop HDFS Issue Type: Bug Reporter: Ted Yu >From >https://builds.apache.org/job/Hadoop-Hdfs-trunk/1626/testReport/org.apache.hadoop.hdfs.server.namenode/TestSecondaryNameNodeUpgrade/testChangeNsIDFails/ > : {code} java.util.ConcurrentModificationException: null at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793) at java.util.HashMap$EntryIterator.next(HashMap.java:834) at java.util.HashMap$EntryIterator.next(HashMap.java:832) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.shutdown(FsVolumeImpl.java:251) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.shutdown(FsVolumeList.java:218) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.shutdown(FsDatasetImpl.java:1414) at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:1309) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:1464) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1439) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1423) at org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade.doIt(TestSecondaryNameNodeUpgrade.java:97) at org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade.testChangeNsIDFails(TestSecondaryNameNodeUpgrade.java:116) {code} The above happens when shutdown() is called in parallel to addBlockPool() or shutdownBlockPool(). -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HDFS-5672) TestHASafeMode#testSafeBlockTracking fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HDFS-5672. -- Resolution: Cannot Reproduce > TestHASafeMode#testSafeBlockTracking fails in trunk > --- > > Key: HDFS-5672 > URL: https://issues.apache.org/jira/browse/HDFS-5672 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Ted Yu > > From build #1614: > {code} > TestHASafeMode.testSafeBlockTracking:623->assertSafeMode:488 Bad safemode > status: 'Safe mode is ON. The reported blocks 3 needs additional 7 blocks to > reach the threshold 0.9990 of total blocks 10. > Safe mode will be turned off automatically' > {code} -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5679) TestCacheDirectives should handle the case where native code is not available
Ted Yu created HDFS-5679: Summary: TestCacheDirectives should handle the case where native code is not available Key: HDFS-5679 URL: https://issues.apache.org/jira/browse/HDFS-5679 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu TestCacheDirectives fails on trunk due to: {code} testBasicPoolOperations(org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives) Time elapsed: 1.618 sec <<< ERROR! java.lang.RuntimeException: Cannot start datanode because the configured max locked memory size (dfs.datanode.max.locked.memory) is greater than zero and native code is not available. {code} Configuration of max locked memory size should be dependent on whether native code is available. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Created] (HDFS-5672) TestHASafeMode#testSafeBlockTracking fails in trunk
Ted Yu created HDFS-5672: Summary: TestHASafeMode#testSafeBlockTracking fails in trunk Key: HDFS-5672 URL: https://issues.apache.org/jira/browse/HDFS-5672 Project: Hadoop HDFS Issue Type: Test Reporter: Ted Yu >From build #1614: {code} TestHASafeMode.testSafeBlockTracking:623->assertSafeMode:488 Bad safemode status: 'Safe mode is ON. The reported blocks 3 needs additional 7 blocks to reach the threshold 0.9990 of total blocks 10. Safe mode will be turned off automatically' {code} -- This message was sent by Atlassian JIRA (v6.1.4#6159)