from:"Ted Yu \(JIRA\)"

[jira] [Created] (HDFS-13515) NetUtils#connect should log remote address for NoRouteToHostException

2018-04-28 Thread Ted Yu (JIRA)

Ted Yu created HDFS-13515:
-

 Summary: NetUtils#connect should log remote address for 
NoRouteToHostException
 Key: HDFS-13515
 URL: https://issues.apache.org/jira/browse/HDFS-13515
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Ted Yu


{code}
hdfs.BlockReaderFactory: I/O error constructing remote block reader.
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2884)
{code}
In the above stack trace, the remote host was not logged.
This makes troubleshooting a bit hard.

NetUtils#connect should log remote address for NoRouteToHostException .



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13511) Provide specialized exception when block length cannot be obtained

2018-04-27 Thread Ted Yu (JIRA)

Ted Yu created HDFS-13511:
-

 Summary: Provide specialized exception when block length cannot be 
obtained
 Key: HDFS-13511
 URL: https://issues.apache.org/jira/browse/HDFS-13511
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Ted Yu


In downstream project, I saw the following code:
{code}
FSDataInputStream inputStream = hdfs.open(new Path(path));
...
if (options.getRecoverFailedOpen() && dfs != null && 
e.getMessage().toLowerCase()
.startsWith("cannot obtain block length for")) {
{code}
The above tightly depends on the following in DFSInputStream#readBlockLength
{code}
throw new IOException("Cannot obtain block length for " + locatedblock);
{code}
The check based on string matching is brittle in production deployment.

After discussing with [~ste...@apache.org], better approach is to introduce 
specialized IOException, e.g. CannotObtainBlockLengthException so that 
downstream project doesn't have to rely on string matching.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-13206) IllegalStateException: Unable to finalize edits file

2018-02-28 Thread Ted Yu (JIRA)

Ted Yu created HDFS-13206:
-

 Summary: IllegalStateException: Unable to finalize edits file
 Key: HDFS-13206
 URL: https://issues.apache.org/jira/browse/HDFS-13206
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Ted Yu


I noticed the following in hbase test output running against hadoop3:
{code}
2018-02-28 18:40:18,491 ERROR [Time-limited test] namenode.JournalSet(402): 
Error: finalize log segment 1, 658 failed for (journal 
JournalAndStream(mgr=FileJournalManager(root=/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1),
 stream=null))
java.lang.IllegalStateException: Unable to finalize edits file 
/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1/current/edits_inprogress_001
  at 
org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:153)
  at 
org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:224)
  at 
org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
  at 
org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:219)
  at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1427)
  at org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:398)
  at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.close(FSEditLogAsync.java:110)
  at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1320)
  at 
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1909)
  at 
org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70)
  at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1013)
  at 
org.apache.hadoop.hdfs.MiniDFSCluster.stopAndJoinNameNode(MiniDFSCluster.java:2047)
  at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1987)
  at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1958)
  at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1951)
  at 
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:767)
  at 
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1109)
  at 
org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport.stopCluster(TestFavoredNodeTableImport.java:71)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-5012) replica.getGenerationStamp() may be >= recoveryId

2017-11-04 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-5012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-5012.
--
Resolution: Cannot Reproduce

> replica.getGenerationStamp() may be >= recoveryId
> -
>
> Key: HDFS-5012
> URL: https://issues.apache.org/jira/browse/HDFS-5012
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.0.5-alpha
>Reporter: Ted Yu
> Attachments: testReplicationQueueFailover.txt
>
>
> The following was first observed by [~jdcryans] in 
> TestReplicationQueueFailover running against 2.0.5-alpha:
> {code}
> 2013-07-16 17:14:33,340 ERROR [IPC Server handler 7 on 35081] 
> security.UserGroupInformation(1481): PriviledgedActionException as:ec2-user 
> (auth:SIMPLE) cause:java.io.IOException: THIS IS NOT SUPPOSED TO HAPPEN: 
> replica.getGenerationStamp() >= recoveryId = 1041, 
> block=blk_4297992342878601848_1041, replica=FinalizedReplica, 
> blk_4297992342878601848_1041, FINALIZED
>   getNumBytes() = 794
>   getBytesOnDisk()  = 794
>   getVisibleLength()= 794
>   getVolume()   = 
> /home/ec2-user/jenkins/workspace/HBase-0.95-Hadoop-2/hbase-server/target/test-data/f2763e32-fe49-4988-ac94-eeca82431821/dfscluster_643a635e-4e39-4aa5-974c-25e01db16ff7/dfs/data/data3/current
>   getBlockFile()= 
> /home/ec2-user/jenkins/workspace/HBase-0.95-Hadoop-2/hbase-server/target/test-data/f2763e32-fe49-4988-ac94-eeca82431821/dfscluster_643a635e-4e39-4aa5-974c-25e01db16ff7/dfs/data/data3/current/BP-1477359609-10.197.55.49-1373994849464/current/finalized/blk_4297992342878601848
>   unlinked  =false
> 2013-07-16 17:14:33,341 WARN  
> [org.apache.hadoop.hdfs.server.datanode.DataNode$2@64a1fcba] 
> datanode.DataNode(1894): Failed to obtain replica info for block 
> (=BP-1477359609-10.197.55.49-1373994849464:blk_4297992342878601848_1041) from 
> datanode (=127.0.0.1:47006)
> java.io.IOException: THIS IS NOT SUPPOSED TO HAPPEN: 
> replica.getGenerationStamp() >= recoveryId = 1041, 
> block=blk_4297992342878601848_1041, replica=FinalizedReplica, 
> blk_4297992342878601848_1041, FINALIZED
>   getNumBytes() = 794
>   getBytesOnDisk()  = 794
>   getVisibleLength()= 794
>   getVolume()   = 
> /home/ec2-user/jenkins/workspace/HBase-0.95-Hadoop-2/hbase-server/target/test-data/f2763e32-fe49-4988-ac94-eeca82431821/dfscluster_643a635e-4e39-4aa5-974c-25e01db16ff7/dfs/data/data3/current
>   getBlockFile()= 
> /home/ec2-user/jenkins/workspace/HBase-0.95-Hadoop-2/hbase-server/target/test-data/f2763e32-fe49-4988-ac94-eeca82431821/dfscluster_643a635e-4e39-4aa5-974c-25e01db16ff7/dfs/data/data3/current/BP-1477359609-10.197.55.49-1373994849464/current/finalized/blk_4297992342878601848
>   unlinked  =false
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-4796) Port HDFS-4721 'Speed up lease/block recovery when DN fails and a block goes into recovery' to branch 1

2017-10-14 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-4796.
--
Resolution: Won't Fix

> Port HDFS-4721 'Speed up lease/block recovery when DN fails and a block goes 
> into recovery' to branch 1
> ---
>
> Key: HDFS-4796
> URL: https://issues.apache.org/jira/browse/HDFS-4796
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Ted Yu
>
> This was observed while doing HBase WAL recovery. HBase uses append to write 
> to its write ahead log. So initially the pipeline is setup as
> DN1 --> DN2 --> DN3
> This WAL needs to be read when DN1 fails since it houses the HBase 
> regionserver for the WAL.
> HBase first recovers the lease on the WAL file. During recovery, we choose 
> DN1 as the primary DN to do the recovery even though DN1 has failed and is 
> not heartbeating any more.
> To speedup lease/block recovery, we always choose the datanode with the most 
> recent heartbeat.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-5834) TestCheckpoint#testCheckpoint may fail due to Bad value assertion

2017-10-10 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-5834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-5834.
--
Resolution: Cannot Reproduce

> TestCheckpoint#testCheckpoint may fail due to Bad value assertion
> -
>
> Key: HDFS-5834
> URL: https://issues.apache.org/jira/browse/HDFS-5834
> Project: Hadoop HDFS
>  Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
>
> I saw the following when running test suite on Linux:
> {code}
> testCheckpoint(org.apache.hadoop.hdfs.server.namenode.TestCheckpoint)  Time 
> elapsed: 3.058 sec  <<< FAILURE!
> java.lang.AssertionError: Bad value for metric GetImageNumOps
> Expected: gt(0)
>  got: <0L>
> at org.junit.Assert.assertThat(Assert.java:780)
> at 
> org.apache.hadoop.test.MetricsAsserts.assertCounterGt(MetricsAsserts.java:318)
> at 
> org.apache.hadoop.hdfs.server.namenode.TestCheckpoint.testCheckpoint(TestCheckpoint.java:1058)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-5718) TestHttpsFileSystem intermittently fails with Port in use error

2017-10-10 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-5718.
--
Resolution: Cannot Reproduce

> TestHttpsFileSystem intermittently fails with Port in use error
> ---
>
> Key: HDFS-5718
> URL: https://issues.apache.org/jira/browse/HDFS-5718
> Project: Hadoop HDFS
>  Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
>
> From 
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1634/testReport/junit/org.apache.hadoop.hdfs.web/TestHttpsFileSystem/org_apache_hadoop_hdfs_web_TestHttpsFileSystem/
>  :
> {code}
> java.net.BindException: Port in use: localhost:50475
>   at java.net.PlainSocketImpl.socketBind(Native Method)
>   at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383)
>   at java.net.ServerSocket.bind(ServerSocket.java:328)
>   at java.net.ServerSocket.(ServerSocket.java:194)
>   at javax.net.ssl.SSLServerSocket.(SSLServerSocket.java:106)
>   at 
> com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.(SSLServerSocketImpl.java:108)
>   at 
> com.sun.net.ssl.internal.ssl.SSLServerSocketFactoryImpl.createServerSocket(SSLServerSocketFactoryImpl.java:72)
>   at 
> org.mortbay.jetty.security.SslSocketConnector.newServerSocket(SslSocketConnector.java:478)
>   at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)
>   at org.apache.hadoop.http.HttpServer.openListeners(HttpServer.java:973)
>   at org.apache.hadoop.http.HttpServer.start(HttpServer.java:914)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:412)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:769)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:315)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1846)
>   at 
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1746)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1203)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:673)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:342)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:323)
>   at 
> org.apache.hadoop.hdfs.web.TestHttpsFileSystem.setUp(TestHttpsFileSystem.java:64)
> {code}
> This could have been caused by concurrent test(s).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-5831) TestAuditLogs#testAuditAllowedStat sometimes fails in trunk

2017-10-10 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-5831.
--
Resolution: Cannot Reproduce

> TestAuditLogs#testAuditAllowedStat sometimes fails in trunk
> ---
>
> Key: HDFS-5831
> URL: https://issues.apache.org/jira/browse/HDFS-5831
> Project: Hadoop HDFS
>  Issue Type: Test
>Reporter: Ted Yu
> Attachments: 
> 5831-org.apache.hadoop.hdfs.server.namenode.TestAuditLogs-output.txt
>
>
> Running TestAuditLogs on Linux, I got:
> {code}
> testAuditAllowedStat[1](org.apache.hadoop.hdfs.server.namenode.TestAuditLogs) 
>  Time elapsed: 6.677 sec  <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:92)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at org.junit.Assert.assertNotNull(Assert.java:526)
> at org.junit.Assert.assertNotNull(Assert.java:537)
> at 
> org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.verifyAuditLogsRepeat(TestAuditLogs.java:312)
> at 
> org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.verifyAuditLogs(TestAuditLogs.java:295)
> at 
> org.apache.hadoop.hdfs.server.namenode.TestAuditLogs.testAuditAllowedStat(TestAuditLogs.java:163)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12599) Move DataNodeTestUtils.mockDatanodeBlkPinning into mock test util class

2017-10-05 Thread Ted Yu (JIRA)

Ted Yu created HDFS-12599:
-

 Summary: Move DataNodeTestUtils.mockDatanodeBlkPinning into mock 
test util class
 Key: HDFS-12599
 URL: https://issues.apache.org/jira/browse/HDFS-12599
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu


HDFS-11164 introduced {{DataNodeTestUtils.mockDatanodeBlkPinning}} which 
brought dependency on mockito back into DataNodeTestUtils

Downstream, this resulted in:
{code}
java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
  at org.apache.hadoop.hdfs.MiniDFSCluster.shouldWait(MiniDFSCluster.java:2668)
  at org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:2564)
  at org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:2607)
  at 
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1667)
  at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:874)
  at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:769)
  at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:661)
  at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1075)
  at 
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:953)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-9169) TestNativeAzureFileSystemOperationsMocked fails in trunk

2015-09-28 Thread Ted Yu (JIRA)

Ted Yu created HDFS-9169:


 Summary: TestNativeAzureFileSystemOperationsMocked fails in trunk
 Key: HDFS-9169
 URL: https://issues.apache.org/jira/browse/HDFS-9169
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


When working on HDFS-6264, QA bot reported the following:
{code}
testGlobStatusFilterWithMultiplePathWildcardsAndNonTrivialFilter(org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked)
  Time elapsed: 0.02 sec  <<< ERROR!
java.lang.NullPointerException: null
at org.apache.hadoop.fs.Globber.glob(Globber.java:145)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1688)
at 
org.apache.hadoop.fs.FSMainOperationsBaseTest.testGlobStatusFilterWithMultiplePathWildcardsAndNonTrivialFilter(FSMainOp
{code}
On hadoop trunk branch, the above can be reproduced without any patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-5897) TestNNWithQJM#testNewNamenodeTakesOverWriter occasionally fails in trunk

2015-09-21 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-5897.
--
Resolution: Cannot Reproduce

> TestNNWithQJM#testNewNamenodeTakesOverWriter occasionally fails in trunk
> 
>
> Key: HDFS-5897
> URL: https://issues.apache.org/jira/browse/HDFS-5897
> Project: Hadoop HDFS
>  Issue Type: Test
>Reporter: Ted Yu
> Attachments: 5897-output.html
>
>
> From 
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1665/testReport/junit/org.apache.hadoop.hdfs.qjournal/TestNNWithQJM/testNewNamenodeTakesOverWriter/
>  :
> {code}
> java.lang.Exception: test timed out after 3 milliseconds
>   at java.net.SocketInputStream.socketRead0(Native Method)
>   at java.net.SocketInputStream.read(SocketInputStream.java:129)
>   at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>   at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>   at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>   at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
>   at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
>   at 
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195)
>   at 
> java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
>   at 
> org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:412)
>   at 
> org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:401)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:396)
> {code}
> I saw:
> {code}
> 2014-02-06 11:38:37,970 ERROR namenode.EditLogInputStream 
> (RedundantEditLogInputStream.java:nextOp(221)) - Got error reading edit log 
> input stream 
> http://localhost:40509/getJournal?jid=myjournal=3=-51%3A1571339494%3A0%3AtestClusterID;
>  failing over to edit log 
> http://localhost:56244/getJournal?jid=myjournal=3=-51%3A1571339494%3A0%3AtestClusterID
> org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException:
>  got premature end-of-file at txid 0; expected file to go up to 4
>   at 
> org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
>   at 
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:83)
>   at 
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:140)
>   at 
> org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
>   at 
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:83)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:167)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:120)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:708)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:606)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:263)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:874)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:634)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:446)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:502)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:658)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:643)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1291)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:939)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:824)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:678)
>   at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:359)
>   at 
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:340)
>   at 
> org.apache.hadoop.hdfs.qjournal.TestNNWithQJM.testNewNamenodeTakesOverWriter(TestNNWithQJM.java:145)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
>

[jira] [Resolved] (HDFS-6037) TestIncrementalBlockReports#testReplaceReceivedBlock fails occasionally in trunk

2015-03-07 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-6037.
--
Resolution: Cannot Reproduce

 TestIncrementalBlockReports#testReplaceReceivedBlock fails occasionally in 
 trunk
 

 Key: HDFS-6037
 URL: https://issues.apache.org/jira/browse/HDFS-6037
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu

 From 
 https://builds.apache.org/job/Hadoop-Hdfs-trunk/1688/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestIncrementalBlockReports/testReplaceReceivedBlock/
  :
 {code}
 datanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(
 any,
 any,
 any
 );
 Wanted 1 time:
 - at 
 org.apache.hadoop.hdfs.server.datanode.TestIncrementalBlockReports.testReplaceReceivedBlock(TestIncrementalBlockReports.java:198)
 But was 2 times. Undesired invocation:
 - at 
 org.apache.hadoop.hdfs.server.datanode.BPServiceActor.reportReceivedDeletedBlocks(BPServiceActor.java:303)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-6081) TestRetryCacheWithHA#testCreateSymlink occasionally fails in trunk

2015-02-15 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-6081.
--
Resolution: Cannot Reproduce

 TestRetryCacheWithHA#testCreateSymlink occasionally fails in trunk
 --

 Key: HDFS-6081
 URL: https://issues.apache.org/jira/browse/HDFS-6081
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu

 From 
 https://builds.apache.org/job/Hadoop-Hdfs-trunk/1696/testReport/junit/org.apache.hadoop.hdfs.server.namenode.ha/TestRetryCacheWithHA/testCreateSymlink/
  :
 {code}
 2014-03-09 13:18:47,515 WARN  security.UserGroupInformation 
 (UserGroupInformation.java:doAs(1600)) - PriviledgedActionException 
 as:jenkins (auth:SIMPLE) cause:java.io.IOException: failed to create link 
 /testlink either because the filename is invalid or the file exists
 2014-03-09 13:18:47,515 INFO  ipc.Server (Server.java:run(2093)) - IPC Server 
 handler 0 on 39303, call 
 org.apache.hadoop.hdfs.protocol.ClientProtocol.createSymlink from 
 127.0.0.1:32909 Call#682 Retry#1: error: java.io.IOException: failed to 
 create link /testlink either because the filename is invalid or the file 
 exists
 java.io.IOException: failed to create link /testlink either because the 
 filename is invalid or the file exists
   at 
 org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlinkInt(FSNamesystem.java:2053)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlink(FSNamesystem.java:2023)
   at 
 org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.createSymlink(NameNodeRpcServer.java:965)
   at 
 org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.createSymlink(ClientNamenodeProtocolServerSideTranslatorPB.java:844)
   at 
 org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
   at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:605)
   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2071)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2067)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1597)
   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2065)
 2014-03-09 13:18:47,522 INFO  blockmanagement.BlockManager 
 (BlockManager.java:processMisReplicatesAsync(2475)) - Total number of blocks  
   = 1
 2014-03-09 13:18:47,523 INFO  blockmanagement.BlockManager 
 (BlockManager.java:processMisReplicatesAsync(2476)) - Number of invalid 
 blocks  = 0
 2014-03-09 13:18:47,523 INFO  blockmanagement.BlockManager 
 (BlockManager.java:processMisReplicatesAsync(2477)) - Number of 
 under-replicated blocks = 0
 2014-03-09 13:18:47,523 INFO  ha.TestRetryCacheWithHA 
 (TestRetryCacheWithHA.java:run(1162)) - Got Exception while calling 
 createSymlink
 org.apache.hadoop.ipc.RemoteException(java.io.IOException): failed to create 
 link /testlink either because the filename is invalid or the file exists
   at 
 org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlinkInt(FSNamesystem.java:2053)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlink(FSNamesystem.java:2023)
   at 
 org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.createSymlink(NameNodeRpcServer.java:965)
   at 
 org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.createSymlink(ClientNamenodeProtocolServerSideTranslatorPB.java:844)
   at 
 org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
   at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:605)
   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2071)
   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2067)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1597)
   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2065)
   at org.apache.hadoop.ipc.Client.call(Client.java:1409)
   at org.apache.hadoop.ipc.Client.call(Client.java:1362)
   at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
   at $Proxy17.createSymlink(Unknown Source)
   at

[jira] [Resolved] (HDFS-6501) TestCrcCorruption#testCorruptionDuringWrt sometimes fails in trunk

2015-02-15 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-6501.
--
Resolution: Cannot Reproduce

 TestCrcCorruption#testCorruptionDuringWrt sometimes fails in trunk
 --

 Key: HDFS-6501
 URL: https://issues.apache.org/jira/browse/HDFS-6501
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor

 From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1767/ :
 {code}
 REGRESSION:  org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt
 Error Message:
 test timed out after 5 milliseconds
 Stack Trace:
 java.lang.Exception: test timed out after 5 milliseconds
 at java.lang.Object.wait(Native Method)
 at 
 org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2024)
 at 
 org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2008)
 at 
 org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2107)
 at 
 org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
 at 
 org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:98)
 at 
 org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt(TestCrcCorruption.java:133)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-6177) TestHttpFSServer fails occasionally in trunk

2015-02-15 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-6177.
--
Resolution: Cannot Reproduce

 TestHttpFSServer fails occasionally in trunk
 

 Key: HDFS-6177
 URL: https://issues.apache.org/jira/browse/HDFS-6177
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor

 From https://builds.apache.org/job/Hadoop-hdfs-trunk/1716/consoleFull :
 {code}
 Running org.apache.hadoop.fs.http.server.TestHttpFSServer
 Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.424 sec  
 FAILURE! - in org.apache.hadoop.fs.http.server.TestHttpFSServer
 testDelegationTokenOperations(org.apache.hadoop.fs.http.server.TestHttpFSServer)
   Time elapsed: 0.559 sec   FAILURE!
 java.lang.AssertionError: expected:401 but was:403
   at org.junit.Assert.fail(Assert.java:93)
   at org.junit.Assert.failNotEquals(Assert.java:647)
   at org.junit.Assert.assertEquals(Assert.java:128)
   at org.junit.Assert.assertEquals(Assert.java:472)
   at org.junit.Assert.assertEquals(Assert.java:456)
   at 
 org.apache.hadoop.fs.http.server.TestHttpFSServer.testDelegationTokenOperations(TestHttpFSServer.java:352)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-6726) TestNamenodeCapacityReport fails intermittently

2015-02-15 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-6726.
--
Resolution: Cannot Reproduce

 TestNamenodeCapacityReport fails intermittently
 ---

 Key: HDFS-6726
 URL: https://issues.apache.org/jira/browse/HDFS-6726
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor

 From 
 https://builds.apache.org/job/Hadoop-Hdfs-trunk/1812/testReport/junit/org.apache.hadoop.hdfs.server.namenode/TestNamenodeCapacityReport/testXceiverCount/
  :
 {code}
 java.io.IOException: Unable to close file because the last block does not 
 have enough number of replicas.
   at 
 org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2141)
   at 
 org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2109)
   at 
 org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport.testXceiverCount(TestNamenodeCapacityReport.java:281)
 {code}
 There were multiple occurrences of 'Broken pipe', 'Connection reset by peer' 
 and 'Premature EOF from inputStream' exceptions in test output



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-7102) Null dereference in PacketReceiver#receiveNextPacket()

2015-02-11 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-7102.
--
Resolution: Later

 Null dereference in PacketReceiver#receiveNextPacket()
 --

 Key: HDFS-7102
 URL: https://issues.apache.org/jira/browse/HDFS-7102
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor

 {code}
   public void receiveNextPacket(ReadableByteChannel in) throws IOException {
 doRead(in, null);
 {code}
 doRead() would pass null as second parameter to (line 134):
 {code}
 doReadFully(ch, in, curPacketBuf);
 {code}
 which dereferences it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-7083) TestDecommission#testIncludeByRegistrationName sometimes fails

2015-02-11 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-7083.
--
Resolution: Cannot Reproduce

 TestDecommission#testIncludeByRegistrationName sometimes fails
 --

 Key: HDFS-7083
 URL: https://issues.apache.org/jira/browse/HDFS-7083
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor

 From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1874/ :
 {code}
 REGRESSION:  
 org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName
 Error Message:
 test timed out after 36 milliseconds
 Stack Trace:
 java.lang.Exception: test timed out after 36 milliseconds
 at java.lang.Thread.sleep(Native Method)
 at 
 org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName(TestDecommission.java:957)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7756) DatanodeInfoWithStorage should be tagged Private

2015-02-09 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7756:


 Summary: DatanodeInfoWithStorage should be tagged Private
 Key: HDFS-7756
 URL: https://issues.apache.org/jira/browse/HDFS-7756
 Project: Hadoop HDFS
  Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu


This is related to HDFS-7647

DatanodeInfoWithStorage was introduced in 
org.apache.hadoop.hdfs.server.protocol package whereas its base class, 
DatanodeInfo, is in org.apache.hadoop.hdfs.protocol

DatanodeInfo is tagged @InterfaceAudience.Private
DatanodeInfoWithStorage should have the same tag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-7220) TestDataNodeMetrics fails in trunk

2015-02-08 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-7220.
--
Resolution: Cannot Reproduce

 TestDataNodeMetrics fails in trunk
 --

 Key: HDFS-7220
 URL: https://issues.apache.org/jira/browse/HDFS-7220
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor

 From 
 https://builds.apache.org/job/Hadoop-Hdfs-trunk/1896/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeMetrics/testSendDataPacketMetrics/
  :
 {code}
 java.lang.NoClassDefFoundError: 
 org/apache/hadoop/util/IntrusiveCollection$IntrusiveIterator
   at 
 org.apache.hadoop.util.IntrusiveCollection.iterator(IntrusiveCollection.java:213)
   at 
 org.apache.hadoop.util.IntrusiveCollection.clear(IntrusiveCollection.java:368)
   at 
 org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.clearPendingCachingCommands(DatanodeManager.java:1590)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1262)
   at 
 org.apache.hadoop.hdfs.server.namenode.FSNamesystem.close(FSNamesystem.java:1590)
   at 
 org.apache.hadoop.hdfs.server.namenode.NameNode.stopCommonServices(NameNode.java:658)
   at 
 org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:823)
   at 
 org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1717)
   at 
 org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1696)
   at 
 org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics.testSendDataPacketMetrics(TestDataNodeMetrics.java:94)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-7267) TestBalancer#testUnknownDatanode occasionally fails in trunk

2015-02-08 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-7267.
--
Resolution: Cannot Reproduce

 TestBalancer#testUnknownDatanode occasionally fails in trunk
 

 Key: HDFS-7267
 URL: https://issues.apache.org/jira/browse/HDFS-7267
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor

 In build #1907 (https://builds.apache.org/job/Hadoop-Hdfs-trunk/1907/):
 {code}
 REGRESSION:  
 org.apache.hadoop.hdfs.server.balancer.TestBalancer.testUnknownDatanode
 Error Message:
 expected:0 but was:-3
 Stack Trace:
 java.lang.AssertionError: expected:0 but was:-3
 at org.junit.Assert.fail(Assert.java:88)
 at org.junit.Assert.failNotEquals(Assert.java:743)
 at org.junit.Assert.assertEquals(Assert.java:118)
 at org.junit.Assert.assertEquals(Assert.java:555)
 at org.junit.Assert.assertEquals(Assert.java:542)
 at 
 org.apache.hadoop.hdfs.server.balancer.TestBalancer.testUnknownDatanode(TestBalancer.java:737)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-7576) TestPipelinesFailover#testFailoverRightBeforeCommitSynchronization sometimes fails in Java 8 build

2015-02-05 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-7576.
--
Resolution: Cannot Reproduce

 TestPipelinesFailover#testFailoverRightBeforeCommitSynchronization sometimes 
 fails in Java 8 build
 --

 Key: HDFS-7576
 URL: https://issues.apache.org/jira/browse/HDFS-7576
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor

 From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/54/ :
 {code}
 REGRESSION:  
 org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization
 Error Message:
 test timed out after 3 milliseconds
 Stack Trace:
 java.lang.Exception: test timed out after 3 milliseconds
 at sun.misc.Unsafe.park(Native Method)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
 at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
 at 
 org.apache.hadoop.test.GenericTestUtils$DelayAnswer.waitForCall(GenericTestUtils.java:226)
 at 
 org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization(TestPipelinesFailover.java:386)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-7571) TestCacheDirectives#testExceedsCapacity sometimes fails in trunk

2015-02-05 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-7571.
--
Resolution: Cannot Reproduce

 TestCacheDirectives#testExceedsCapacity sometimes fails in trunk
 

 Key: HDFS-7571
 URL: https://issues.apache.org/jira/browse/HDFS-7571
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor

 From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1985/ :
 {code}
 REGRESSION:  
 org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.testExceedsCapacity
 Error Message:
 Pending cached list of 127.0.0.1:47332 is not empty, [{blockId=1073741841, 
 replication=1, mark=true}]
 Stack Trace:
 java.lang.AssertionError: Pending cached list of 127.0.0.1:47332 is not 
 empty, [{blockId=1073741841, replication=1, mark=true}]
 at org.junit.Assert.fail(Assert.java:88)
 at org.junit.Assert.assertTrue(Assert.java:41)
 at 
 org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.checkPendingCachedEmpty(TestCacheDirectives.java:1420)
 at 
 org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.testExceedsCapacity(TestCacheDirectives.java:1443)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-7464) TestDFSAdminWithHA#testRefreshSuperUserGroupsConfiguration fails against Java 8

2015-01-24 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-7464.
--
Resolution: Cannot Reproduce

 TestDFSAdminWithHA#testRefreshSuperUserGroupsConfiguration fails against Java 
 8
 ---

 Key: HDFS-7464
 URL: https://issues.apache.org/jira/browse/HDFS-7464
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor

 From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/23/ :
 {code}
 REGRESSION:  
 org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA.testRefreshSuperUserGroupsConfiguration
 Error Message:
 refreshSuperUserGroupsConfiguration: End of File Exception between local host 
 is: asf908.gq1.ygridcore.net/67.195.81.152; destination host is: 
 localhost:12700; : java.io.EOFException; For more details see:  
 http://wiki.apache.org/hadoop/EOFException expected:0 but was:-1
 Stack Trace:
 java.lang.AssertionError: refreshSuperUserGroupsConfiguration: End of File 
 Exception between local host is: asf908.gq1.ygridcore.net/67.195.81.152; 
 destination host is: localhost:12700; : java.io.EOFException; For more 
 details see:  http://wiki.apache.org/hadoop/EOFException expected:0 but 
 was:-1
 at org.junit.Assert.fail(Assert.java:88)
 at org.junit.Assert.failNotEquals(Assert.java:743)
 at org.junit.Assert.assertEquals(Assert.java:118)
 at org.junit.Assert.assertEquals(Assert.java:555)
 at 
 org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA.testRefreshSuperUserGroupsConfiguration(TestDFSAdminWithHA.java:228)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-7422) TestEncryptionZonesWithKMS fails against Java 8

2015-01-24 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-7422.
--
Resolution: Cannot Reproduce

 TestEncryptionZonesWithKMS fails against Java 8
 ---

 Key: HDFS-7422
 URL: https://issues.apache.org/jira/browse/HDFS-7422
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor

 From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/12/ :
 {code}
 REGRESSION:  
 org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS.testReadWriteUsingWebHdfs
 Error Message:
 Stream closed.
 Stack Trace:
 java.io.IOException: Stream closed.
 at sun.reflect.GeneratedConstructorAccessor58.newInstance(Unknown 
 Source)
 at 
 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at 
 org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
 at 
 org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
 at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:385)
 at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:91)
 at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:656)
 at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:622)
 at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
 at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
 at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
 at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$UnresolvedUrlOpener.connect(WebHdfsFileSystem.java:1204)
 at 
 org.apache.hadoop.hdfs.web.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:120)
 at 
 org.apache.hadoop.hdfs.web.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:104)
 at 
 org.apache.hadoop.hdfs.web.ByteRangeInputStream.init(ByteRangeInputStream.java:89)
 at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$OffsetUrlInputStream.init(WebHdfsFileSystem.java:1261)
 at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem.open(WebHdfsFileSystem.java:1175)
 at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
 at 
 org.apache.hadoop.hdfs.DFSTestUtil.verifyFilesEqual(DFSTestUtil.java:1399)
 at 
 org.apache.hadoop.hdfs.TestEncryptionZones.testReadWriteUsingWebHdfs(TestEncryptionZones.java:634)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at 
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
 at 
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at 
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
 at 
 org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at 
 org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
 Caused by: org.apache.hadoop.ipc.RemoteException: Stream closed.
 at 
 org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:165)
 at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:353)
 at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
 at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608)
 at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
 at 
 org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
 at

[jira] [Resolved] (HDFS-7311) TestLeaseRecovery2 sometimes fails in trunk

2015-01-24 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-7311.
--
Resolution: Cannot Reproduce

 TestLeaseRecovery2 sometimes fails in trunk
 ---

 Key: HDFS-7311
 URL: https://issues.apache.org/jira/browse/HDFS-7311
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor

 From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1917/ :
 {code}
 REGRESSION:  org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery
 Error Message:
 Call From asf909.gq1.ygridcore.net/67.195.81.153 to localhost:55061 failed on 
 connection exception: java.net.ConnectException: Connection refused; For more 
 details see:  http://wiki.apache.org/hadoop/ConnectionRefused
 Stack Trace:
 java.net.ConnectException: Call From asf909.gq1.ygridcore.net/67.195.81.153 
 to localhost:55061 failed on connection exception: java.net.ConnectException: 
 Connection refused; For more details see:  
 http://wiki.apache.org/hadoop/ConnectionRefused
 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
 at 
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
 at 
 org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
 at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
 at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
 at 
 org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
 at 
 org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
 at 
 org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
 at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
 at org.apache.hadoop.ipc.Client.call(Client.java:1438)
 at org.apache.hadoop.ipc.Client.call(Client.java:1399)
 at 
 org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
 at com.sun.proxy.$Proxy19.create(Unknown Source)
 at 
 org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:295)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at 
 org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
 at 
 org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
 at com.sun.proxy.$Proxy20.create(Unknown Source)
 at 
 org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1694)
 at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1654)
 at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1579)
 at 
 org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:397)
 at 
 org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:393)
 at 
 org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
 at 
 org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:393)
 at 
 org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:337)
 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
 at 
 org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery(TestLeaseRecovery2.java:276)
 FAILED:  
 org.apache.hadoop.hdfs.TestLeaseRecovery2.org.apache.hadoop.hdfs.TestLeaseRecovery2
 Error Message:
 Test resulted in an unexpected exit
 Stack Trace:
 java.lang.AssertionError: Test resulted in an unexpected exit
 at 
 org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1709)
 at 
 org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1696)
 at 
 org.apache.hadoop.hdfs.TestLeaseRecovery2.tearDown(TestLeaseRecovery2.java:105)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7606) Missing null check in INodeFile#getBlocks()

2015-01-13 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7606:


 Summary: Missing null check in INodeFile#getBlocks()
 Key: HDFS-7606
 URL: https://issues.apache.org/jira/browse/HDFS-7606
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
BlockInfo[] snapshotBlocks = diff == null ? getBlocks() : diff.getBlocks();
if(snapshotBlocks != null)
  return snapshotBlocks;
// Blocks are not in the current snapshot
// Find next snapshot with blocks present or return current file blocks
snapshotBlocks = getDiffs().findLaterSnapshotBlocks(diff.getSnapshotId());
{code}
If diff is null and snapshotBlocks is null, NullPointerException would result 
from the call to diff.getSnapshotId().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-7493) removedDst should be checked against null in finally block of FSDirRenameOp#unprotectedRenameTo()

2015-01-01 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-7493.
--
Resolution: Duplicate

Dup of HDFS-7538

 removedDst should be checked against null in finally block of 
 FSDirRenameOp#unprotectedRenameTo()
 -

 Key: HDFS-7493
 URL: https://issues.apache.org/jira/browse/HDFS-7493
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor

 {code}
   removedDst = dstIIP.getLastINode();
   undoRemoveDst = true;
 {code}
 If removedDst is null, the following code in finally block may result in NPE:
 {code}
 if (dstParent.isDirectory() 
 dstParent.asDirectory().isWithSnapshot()) {
   dstParent.asDirectory().undoRename4DstParent(removedDst,
   dstIIP.getLatestSnapshotId());
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7576) TestPipelinesFailover#testFailoverRightBeforeCommitSynchronization sometimes fails in Java 8 build

2014-12-30 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7576:


 Summary: 
TestPipelinesFailover#testFailoverRightBeforeCommitSynchronization sometimes 
fails in Java 8 build
 Key: HDFS-7576
 URL: https://issues.apache.org/jira/browse/HDFS-7576
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/54/ :
{code}
REGRESSION:  
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization

Error Message:
test timed out after 3 milliseconds

Stack Trace:
java.lang.Exception: test timed out after 3 milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at 
org.apache.hadoop.test.GenericTestUtils$DelayAnswer.waitForCall(GenericTestUtils.java:226)
at 
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization(TestPipelinesFailover.java:386)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7571) TestCacheDirectives#testExceedsCapacity sometimes fails in trunk

2014-12-26 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7571:


 Summary: TestCacheDirectives#testExceedsCapacity sometimes fails 
in trunk
 Key: HDFS-7571
 URL: https://issues.apache.org/jira/browse/HDFS-7571
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1985/ :
{code}
REGRESSION:  
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.testExceedsCapacity

Error Message:
Pending cached list of 127.0.0.1:47332 is not empty, [{blockId=1073741841, 
replication=1, mark=true}]

Stack Trace:
java.lang.AssertionError: Pending cached list of 127.0.0.1:47332 is not empty, 
[{blockId=1073741841, replication=1, mark=true}]
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.checkPendingCachedEmpty(TestCacheDirectives.java:1420)
at 
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.testExceedsCapacity(TestCacheDirectives.java:1443)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7567) Potential null dereference in FSEditLogLoader#applyEditLogOp()

2014-12-23 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7567:


 Summary: Potential null dereference in 
FSEditLogLoader#applyEditLogOp()
 Key: HDFS-7567
 URL: https://issues.apache.org/jira/browse/HDFS-7567
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
  INodeFile oldFile = INodeFile.valueOf(iip.getLastINode(), path, true);
  if (oldFile != null  addCloseOp.overwrite) {
...
  INodeFile newFile = oldFile;
...
  // Update the salient file attributes.
  newFile.setAccessTime(addCloseOp.atime, Snapshot.CURRENT_STATE_ID);
  newFile.setModificationTime(addCloseOp.mtime, Snapshot.CURRENT_STATE_ID);
{code}
The last two lines are not protected by null check.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HDFS-7567) Potential null dereference in FSEditLogLoader#applyEditLogOp()

2014-12-23 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reopened HDFS-7567:
--
  Assignee: Ted Yu

 Potential null dereference in FSEditLogLoader#applyEditLogOp()
 --

 Key: HDFS-7567
 URL: https://issues.apache.org/jira/browse/HDFS-7567
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor

 {code}
   INodeFile oldFile = INodeFile.valueOf(iip.getLastINode(), path, true);
   if (oldFile != null  addCloseOp.overwrite) {
 ...
   INodeFile newFile = oldFile;
 ...
   // Update the salient file attributes.
   newFile.setAccessTime(addCloseOp.atime, Snapshot.CURRENT_STATE_ID);
   newFile.setModificationTime(addCloseOp.mtime, 
 Snapshot.CURRENT_STATE_ID);
 {code}
 The last two lines are not protected by null check.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7538) removedDst should be checked against null in the finally block of FSDirRenameOp#unprotectedRenameTo()

2014-12-16 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7538:


 Summary: removedDst should be checked against null in the finally 
block of FSDirRenameOp#unprotectedRenameTo()
 Key: HDFS-7538
 URL: https://issues.apache.org/jira/browse/HDFS-7538
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
if (removedDst != null) {
  undoRemoveDst = false;
...
  if (undoRemoveDst) {
// Rename failed - restore dst
if (dstParent.isDirectory() 
dstParent.asDirectory().isWithSnapshot()) {
  dstParent.asDirectory().undoRename4DstParent(removedDst,
{code}
If the first if check doesn't pass, removedDst would be null and undoRemoveDst 
may be true.
This combination would lead to NullPointerException in the finally block.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7522) TestDecommission#testIncludeByRegistrationName sometimes timeout

2014-12-13 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7522:


 Summary: TestDecommission#testIncludeByRegistrationName sometimes 
timeout
 Key: HDFS-7522
 URL: https://issues.apache.org/jira/browse/HDFS-7522
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From 
https://builds.apache.org/job/Hadoop-hdfs-trunk/lastCompletedBuild/testReport/org.apache.hadoop.hdfs/TestDecommission/testIncludeByRegistrationName/
 :
{code}
java.lang.Exception: test timed out after 36 milliseconds
at java.lang.Thread.sleep(Native Method)
at 
org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName(TestDecommission.java:957)
{code}
In the test output, the following repeated over 300 times:
{code}
2014-12-13 18:44:29,910 ERROR datanode.DataNode (BPServiceActor.java:run(836)) 
- Initialization failed for Block pool BP-408261154-67.195.81.152-1418496249312 
(Datanode Uuid null) service to localhost/127.0.0.1:38607 Datanode denied 
communication with namenode because the host is not in the include-list: 
DatanodeRegistration(127.0.0.1, 
datanodeUuid=755318af-3336-462d-9bd7-2a7b966ee4f4, infoPort=45707, 
infoSecurePort=0, ipcPort=46621, 
storageInfo=lv=-56;cid=testClusterID;nsid=1154823031;c=0)
at 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:915)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:4402)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:1196)
at 
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:92)
at 
org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:26296)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:966)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2127)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2123)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2121)

2014-12-13 18:44:30,871 INFO  hdfs.TestDecommission 
(TestDecommission.java:testIncludeByRegistrationName(956)) - Waiting for 
datanode to come back
2014-12-13 18:44:31,873 INFO  hdfs.TestDecommission 
(TestDecommission.java:testIncludeByRegistrationName(956)) - Waiting for 
datanode to come back
2014-12-13 18:44:32,874 INFO  hdfs.TestDecommission 
(TestDecommission.java:testIncludeByRegistrationName(956)) - Waiting for 
datanode to come back
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7493) removedDst should be checked against null in finally block of FSDirRenameOp#unprotectedRenameTo()

2014-12-08 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7493:


 Summary: removedDst should be checked against null in finally 
block of FSDirRenameOp#unprotectedRenameTo()
 Key: HDFS-7493
 URL: https://issues.apache.org/jira/browse/HDFS-7493
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
  removedDst = dstIIP.getLastINode();
  undoRemoveDst = true;
{code}
If removedDst is null, the following code in finally block may result in NPE:
{code}
if (dstParent.isDirectory() 
dstParent.asDirectory().isWithSnapshot()) {
  dstParent.asDirectory().undoRename4DstParent(removedDst,
  dstIIP.getLatestSnapshotId());
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7494) Checking of closed in DFSInputStream#pread() should be protected by synchronization

2014-12-08 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7494:


 Summary: Checking of closed in DFSInputStream#pread() should be 
protected by synchronization
 Key: HDFS-7494
 URL: https://issues.apache.org/jira/browse/HDFS-7494
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
  private int pread(long position, byte[] buffer, int offset, int length)
  throws IOException {
// sanity checks
dfsClient.checkOpen();
if (closed) {
{code}
Checking of closed should be protected by holding lock on DFSInputStream.this



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7471) TestDatanodeManager#testNumVersionsReportedCorrect occasionally fails

2014-12-04 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7471:


 Summary: TestDatanodeManager#testNumVersionsReportedCorrect 
occasionally fails
 Key: HDFS-7471
 URL: https://issues.apache.org/jira/browse/HDFS-7471
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1957/ :
{code}
FAILED:  
org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager.testNumVersionsReportedCorrect

Error Message:
The map of version counts returned by DatanodeManager was not what it was 
expected to be on iteration 237 expected:0 but was:1

Stack Trace:
java.lang.AssertionError: The map of version counts returned by DatanodeManager 
was not what it was expected to be on iteration 237 expected:0 but was:1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at 
org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager.testNumVersionsReportedCorrect(TestDatanodeManager.java:150)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7464) TestDFSAdminWithHA#testRefreshSuperUserGroupsConfiguration fails against Java 8

2014-12-02 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7464:


 Summary: 
TestDFSAdminWithHA#testRefreshSuperUserGroupsConfiguration fails against Java 8
 Key: HDFS-7464
 URL: https://issues.apache.org/jira/browse/HDFS-7464
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/23/ :
{code}
REGRESSION:  
org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA.testRefreshSuperUserGroupsConfiguration

Error Message:
refreshSuperUserGroupsConfiguration: End of File Exception between local host 
is: asf908.gq1.ygridcore.net/67.195.81.152; destination host is: 
localhost:12700; : java.io.EOFException; For more details see:  
http://wiki.apache.org/hadoop/EOFException expected:0 but was:-1

Stack Trace:
java.lang.AssertionError: refreshSuperUserGroupsConfiguration: End of File 
Exception between local host is: asf908.gq1.ygridcore.net/67.195.81.152; 
destination host is: localhost:12700; : java.io.EOFException; For more 
details see:  http://wiki.apache.org/hadoop/EOFException expected:0 but 
was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at 
org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA.testRefreshSuperUserGroupsConfiguration(TestDFSAdminWithHA.java:228)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7455) TestReplaceDatanodeOnFailure sometimes fails

2014-11-29 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7455:


 Summary: TestReplaceDatanodeOnFailure sometimes fails
 Key: HDFS-7455
 URL: https://issues.apache.org/jira/browse/HDFS-7455
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1947/ :
{code}
REGRESSION:  
org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure.testReplaceDatanodeOnFailure

Error Message:
expected:3 but was:2

Stack Trace:
java.lang.AssertionError: expected:3 but was:2
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure$SlowWriter.checkReplication(TestReplaceDatanodeOnFailure.java:235)
at 
org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure.testReplaceDatanodeOnFailure(TestReplaceDatanodeOnFailure.java:154)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7448) TestBookKeeperHACheckpoints fails in trunk build

2014-11-26 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7448:


 Summary: TestBookKeeperHACheckpoints fails in trunk build
 Key: HDFS-7448
 URL: https://issues.apache.org/jira/browse/HDFS-7448
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


The test failed against both java 7 and java 8.
From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/17/console :
{code}
testStandbyExceptionThrownDuringCheckpoint(org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints)
  Time elapsed: 6.822 sec   ERROR!
org.apache.hadoop.ipc.RemoteException: File /testFile could only be replicated 
to 0 nodes instead of minReplication (=1).  There are 0 datanode(s) running and 
no node(s) are excluded in this operation.
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1558)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3024)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:699)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:966)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2125)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2121)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2119)

at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy20.addBlock(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy21.addBlock(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1544)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:600)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7422) TestEncryptionZonesWithKMS fails against Java 8

2014-11-21 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7422:


 Summary: TestEncryptionZonesWithKMS fails against Java 8
 Key: HDFS-7422
 URL: https://issues.apache.org/jira/browse/HDFS-7422
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/12/ :
{code}
REGRESSION:  
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS.testReadWriteUsingWebHdfs

Error Message:
Stream closed.

Stack Trace:
java.io.IOException: Stream closed.
at sun.reflect.GeneratedConstructorAccessor58.newInstance(Unknown 
Source)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at 
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at 
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:385)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:91)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:656)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:622)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$UnresolvedUrlOpener.connect(WebHdfsFileSystem.java:1204)
at 
org.apache.hadoop.hdfs.web.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:120)
at 
org.apache.hadoop.hdfs.web.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:104)
at 
org.apache.hadoop.hdfs.web.ByteRangeInputStream.init(ByteRangeInputStream.java:89)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$OffsetUrlInputStream.init(WebHdfsFileSystem.java:1261)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.open(WebHdfsFileSystem.java:1175)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at 
org.apache.hadoop.hdfs.DFSTestUtil.verifyFilesEqual(DFSTestUtil.java:1399)
at 
org.apache.hadoop.hdfs.TestEncryptionZones.testReadWriteUsingWebHdfs(TestEncryptionZones.java:634)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: org.apache.hadoop.ipc.RemoteException: Stream closed.
at 
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:165)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:353)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$UnresolvedUrlOpener.connect(WebHdfsFileSystem.java:1204)
at 
org.apache.hadoop.hdfs.web.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:120)
at

[jira] [Created] (HDFS-7399) Lack of synchronization in DFSOutputStream#Packet#getLastByteOffsetBlock()

2014-11-14 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7399:


 Summary: Lack of synchronization in 
DFSOutputStream#Packet#getLastByteOffsetBlock()
 Key: HDFS-7399
 URL: https://issues.apache.org/jira/browse/HDFS-7399
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
long getLastByteOffsetBlock() {
  return offsetInBlock + dataPos - dataStart;
{code}
Access to fields of Packet.this should be protected by synchronization as done 
in other methods such as writeTo().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7393) TestDFSUpgradeFromImage#testUpgradeFromCorruptRel22Image fails in trunk

2014-11-12 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7393:


 Summary: TestDFSUpgradeFromImage#testUpgradeFromCorruptRel22Image 
fails in trunk
 Key: HDFS-7393
 URL: https://issues.apache.org/jira/browse/HDFS-7393
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


The following is reproducible:
{code}
Running org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.017 sec  
FAILURE! - in org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
testUpgradeFromCorruptRel22Image(org.apache.hadoop.hdfs.TestDFSUpgradeFromImage)
  Time elapsed: 1.005 sec   ERROR!
java.lang.IllegalStateException: null
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockIdManager.setGenerationStampV1Limit(BlockIdManager.java:85)
at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockIdManager.clear(BlockIdManager.java:206)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.clear(FSNamesystem.java:622)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:667)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:376)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:268)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:991)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:537)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:596)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:763)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:747)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1443)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1104)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:975)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:804)
at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:465)
at 
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:424)
at 
org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.upgradeAndVerify(TestDFSUpgradeFromImage.java:582)
at 
org.apache.hadoop.hdfs.TestDFSUpgradeFromImage.testUpgradeFromCorruptRel22Image(TestDFSUpgradeFromImage.java:318)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7311) TestLeaseRecovery2 sometimes fails in trunk

2014-10-30 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7311:


 Summary: TestLeaseRecovery2 sometimes fails in trunk
 Key: HDFS-7311
 URL: https://issues.apache.org/jira/browse/HDFS-7311
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1917/ :
{code}
REGRESSION:  org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery

Error Message:
Call From asf909.gq1.ygridcore.net/67.195.81.153 to localhost:55061 failed on 
connection exception: java.net.ConnectException: Connection refused; For more 
details see:  http://wiki.apache.org/hadoop/ConnectionRefused

Stack Trace:
java.net.ConnectException: Call From asf909.gq1.ygridcore.net/67.195.81.153 to 
localhost:55061 failed on connection exception: java.net.ConnectException: 
Connection refused; For more details see:  
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy19.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:295)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy20.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1694)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1654)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1579)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:397)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$6.doCall(DistributedFileSystem.java:393)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:393)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:337)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
at 
org.apache.hadoop.hdfs.TestLeaseRecovery2.testHardLeaseRecovery(TestLeaseRecovery2.java:276)


FAILED:  
org.apache.hadoop.hdfs.TestLeaseRecovery2.org.apache.hadoop.hdfs.TestLeaseRecovery2

Error Message:
Test resulted in an unexpected exit

Stack Trace:
java.lang.AssertionError: Test resulted in an unexpected exit
at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1709)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1696)
at 
org.apache.hadoop.hdfs.TestLeaseRecovery2.tearDown(TestLeaseRecovery2.java:105)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7289) TestDFSUpgradeWithHA sometimes fails in trunk

2014-10-25 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7289:


 Summary: TestDFSUpgradeWithHA sometimes fails in trunk
 Key: HDFS-7289
 URL: https://issues.apache.org/jira/browse/HDFS-7289
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From trunk build #1912:
{code}
REGRESSION:  
org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA.testFinalizeFromSecondNameNodeWithJournalNodes

Error Message:
java.lang.RuntimeException: java.net.SocketTimeoutException: Read timed out

Stack Trace:
java.io.IOException: java.lang.RuntimeException: 
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:698)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:641)
at 
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1218)
at 
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
at 
org.apache.hadoop.hdfs.server.namenode.TransferFsImage.doGetUrl(TransferFsImage.java:410)
at 
org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:395)
at 
org.apache.hadoop.hdfs.server.namenode.TransferFsImage.downloadImageToStorage(TransferFsImage.java:114)
at 
org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.doRun(BootstrapStandby.java:213)
at 
org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.access$000(BootstrapStandby.java:69)
at 
org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby$1.run(BootstrapStandby.java:107)
at 
org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby$1.run(BootstrapStandby.java:103)
at 
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:414)
at 
org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at 
org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:315)
at 
org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA.testFinalizeFromSecondNameNodeWithJournalNodes(TestDFSUpgradeWithHA.java:493)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7267) TestBalancer#testUnknownDatanode occasionally fails in trunk

2014-10-20 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7267:


 Summary: TestBalancer#testUnknownDatanode occasionally fails in 
trunk
 Key: HDFS-7267
 URL: https://issues.apache.org/jira/browse/HDFS-7267
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


In build #1907:
{code}
REGRESSION:  
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testUnknownDatanode

Error Message:
expected:0 but was:-3

Stack Trace:
java.lang.AssertionError: expected:0 but was:-3
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testUnknownDatanode(TestBalancer.java:737)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7261) storageMap is accessed without synchronization in DatanodeDescriptor#updateHeartbeatState()

2014-10-17 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7261:


 Summary: storageMap is accessed without synchronization in 
DatanodeDescriptor#updateHeartbeatState()
 Key: HDFS-7261
 URL: https://issues.apache.org/jira/browse/HDFS-7261
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


Here is the code:
{code}
  failedStorageInfos = new HashSetDatanodeStorageInfo(
  storageMap.values());
{code}
In other places, the lock on DatanodeDescriptor.storageMap is held:
{code}
synchronized (storageMap) {
  final CollectionDatanodeStorageInfo storages = storageMap.values();
  return storages.toArray(new DatanodeStorageInfo[storages.size()]);
}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7220) TestDataNodeMetrics fails in trunk

2014-10-09 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7220:


 Summary: TestDataNodeMetrics fails in trunk
 Key: HDFS-7220
 URL: https://issues.apache.org/jira/browse/HDFS-7220
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From 
https://builds.apache.org/job/Hadoop-Hdfs-trunk/1896/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeMetrics/testSendDataPacketMetrics/
 :
{code}
java.lang.NoClassDefFoundError: 
org/apache/hadoop/util/IntrusiveCollection$IntrusiveIterator
at 
org.apache.hadoop.util.IntrusiveCollection.iterator(IntrusiveCollection.java:213)
at 
org.apache.hadoop.util.IntrusiveCollection.clear(IntrusiveCollection.java:368)
at 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.clearPendingCachingCommands(DatanodeManager.java:1590)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1262)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.close(FSNamesystem.java:1590)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.stopCommonServices(NameNode.java:658)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:823)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1717)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1696)
at 
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics.testSendDataPacketMetrics(TestDataNodeMetrics.java:94)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-7103) TestStorageMover sometimes fails in trunk

2014-09-22 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-7103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-7103.
--
Resolution: Duplicate

Dup of HDFS-7095

 TestStorageMover sometimes fails in trunk
 -

 Key: HDFS-7103
 URL: https://issues.apache.org/jira/browse/HDFS-7103
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor

 From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1876/console :
 {code}
 Running org.apache.hadoop.hdfs.server.mover.TestStorageMover
 Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 125.179 sec 
  FAILURE! - in org.apache.hadoop.hdfs.server.mover.TestStorageMover
 testNoSpaceArchive(org.apache.hadoop.hdfs.server.mover.TestStorageMover)  
 Time elapsed: 11.695 sec   FAILURE!
 java.lang.AssertionError: null
   at org.junit.Assert.fail(Assert.java:86)
   at org.junit.Assert.assertTrue(Assert.java:41)
   at org.junit.Assert.assertTrue(Assert.java:52)
   at 
 org.apache.hadoop.hdfs.server.mover.TestStorageMover.testNoSpaceArchive(TestStorageMover.java:720)
 testNoSpaceDisk(org.apache.hadoop.hdfs.server.mover.TestStorageMover)  Time 
 elapsed: 11.596 sec   FAILURE!
 java.lang.AssertionError: null
   at org.junit.Assert.fail(Assert.java:86)
   at org.junit.Assert.assertTrue(Assert.java:41)
   at org.junit.Assert.assertTrue(Assert.java:52)
   at 
 org.apache.hadoop.hdfs.server.mover.TestStorageMover.testNoSpaceDisk(TestStorageMover.java:637)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7101) Potential null dereference in DFSck#doWork()

2014-09-19 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7101:


 Summary: Potential null dereference in DFSck#doWork()
 Key: HDFS-7101
 URL: https://issues.apache.org/jira/browse/HDFS-7101
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
String lastLine = null;
int errCode = -1;
try {
  while ((line = input.readLine()) != null) {
...
if (lastLine.endsWith(NamenodeFsck.HEALTHY_STATUS)) {
  errCode = 0;
{code}
If readLine() throws exception, lastLine may be null, leading to NPE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7102) Null dereference in PacketReceiver#receiveNextPacket()

2014-09-19 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7102:


 Summary: Null dereference in PacketReceiver#receiveNextPacket()
 Key: HDFS-7102
 URL: https://issues.apache.org/jira/browse/HDFS-7102
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
  public void receiveNextPacket(ReadableByteChannel in) throws IOException {
doRead(in, null);
{code}
doRead() would pass null as second parameter to (line 134):
{code}
doReadFully(ch, in, curPacketBuf);
{code}
which dereferences it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7103) TestStorageMover sometimes fails in trunk

2014-09-19 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7103:


 Summary: TestStorageMover sometimes fails in trunk
 Key: HDFS-7103
 URL: https://issues.apache.org/jira/browse/HDFS-7103
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1876/console :
{code}
Running org.apache.hadoop.hdfs.server.mover.TestStorageMover
Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 125.179 sec  
FAILURE! - in org.apache.hadoop.hdfs.server.mover.TestStorageMover
testNoSpaceArchive(org.apache.hadoop.hdfs.server.mover.TestStorageMover)  Time 
elapsed: 11.695 sec   FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.hadoop.hdfs.server.mover.TestStorageMover.testNoSpaceArchive(TestStorageMover.java:720)

testNoSpaceDisk(org.apache.hadoop.hdfs.server.mover.TestStorageMover)  Time 
elapsed: 11.596 sec   FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.hadoop.hdfs.server.mover.TestStorageMover.testNoSpaceDisk(TestStorageMover.java:637)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7083) TestDecommission#testIncludeByRegistrationName sometimes fails

2014-09-17 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7083:


 Summary: TestDecommission#testIncludeByRegistrationName sometimes 
fails
 Key: HDFS-7083
 URL: https://issues.apache.org/jira/browse/HDFS-7083
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1874/ :
{code}
REGRESSION:  
org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName

Error Message:
test timed out after 36 milliseconds

Stack Trace:
java.lang.Exception: test timed out after 36 milliseconds
at java.lang.Thread.sleep(Native Method)
at 
org.apache.hadoop.hdfs.TestDecommission.testIncludeByRegistrationName(TestDecommission.java:957)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7008) xlator should be closed upon exit from DFSAdmin#genericRefresh()

2014-09-05 Thread Ted Yu (JIRA)

Ted Yu created HDFS-7008:


 Summary: xlator should be closed upon exit from 
DFSAdmin#genericRefresh()
 Key: HDFS-7008
 URL: https://issues.apache.org/jira/browse/HDFS-7008
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
GenericRefreshProtocol xlator =
  new GenericRefreshProtocolClientSideTranslatorPB(proxy);

// Refresh
CollectionRefreshResponse responses = xlator.refresh(identifier, args);
{code}
GenericRefreshProtocolClientSideTranslatorPB#close() should be called on xlator 
before return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-6902) FileWriter should be closed in finally block in BlockReceiver#receiveBlock()

2014-08-20 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6902:


 Summary: FileWriter should be closed in finally block in 
BlockReceiver#receiveBlock()
 Key: HDFS-6902
 URL: https://issues.apache.org/jira/browse/HDFS-6902
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


Here is code starting from line 828:
{code}
try {
  FileWriter out = new FileWriter(restartMeta);
  // write out the current time.
  out.write(Long.toString(Time.now() + restartBudget));
  out.flush();
  out.close();
} catch (IOException ioe) {
{code}
If write() or flush() call throws IOException, out wouldn't be closed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6852) TestBalancerWithEncryptedTransfer sometimes fails in trunk

2014-08-14 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6852:


 Summary: TestBalancerWithEncryptedTransfer sometimes fails in trunk
 Key: HDFS-6852
 URL: https://issues.apache.org/jira/browse/HDFS-6852
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1837/ :
{code}
REGRESSION:  
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer.testEncryptedBalancer0

Error Message:
Rebalancing expected avg utilization to become 0.2, but on datanode 
127.0.0.1:47596 it remains at 0.08 after more than 4 msec.

Stack Trace:
java.util.concurrent.TimeoutException: Rebalancing expected avg utilization to 
become 0.2, but on datanode 127.0.0.1:47596 it remains at 0.08 after more than 
4 msec.
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.waitForBalancer(TestBalancer.java:321)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.runBalancer(TestBalancer.java:580)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:551)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:437)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.oneNodeTest(TestBalancer.java:645)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancer0Internal(TestBalancer.java:759)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer.testEncryptedBalancer0(TestBalancerWithEncryptedTransfer.java:38)
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6848) Lack of synchronization on access to datanodeUuid in DataStorage#format()

2014-08-13 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6848:


 Summary: Lack of synchronization on access to datanodeUuid in 
DataStorage#format() 
 Key: HDFS-6848
 URL: https://issues.apache.org/jira/browse/HDFS-6848
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
this.datanodeUuid = datanodeUuid;
{code}
The above assignment should be done holding lock DataStorage.this - as is 
done in two other places.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6842) TestHttpFSFWithWebhdfsFileSystem fails in trunk

2014-08-12 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6842:


 Summary: TestHttpFSFWithWebhdfsFileSystem fails in trunk
 Key: HDFS-6842
 URL: https://issues.apache.org/jira/browse/HDFS-6842
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu


This can be reproduced locally:
{code}
testOperationDoAs[21](org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem)
  Time elapsed: 0.315 sec   ERROR!
org.apache.hadoop.ipc.RemoteException: User: zy is not allowed to impersonate 
user1
at org.apache.hadoop.ipc.Client.call(Client.java:1411)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy24.mkdirs(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:512)
at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy25.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2546)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2517)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:821)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:817)
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6810) storageMap is accessed without proper synchronization in DatanodeDescriptor#getStorageReports

2014-08-02 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6810:


 Summary: storageMap is accessed without proper synchronization in 
DatanodeDescriptor#getStorageReports
 Key: HDFS-6810
 URL: https://issues.apache.org/jira/browse/HDFS-6810
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


Here is related code:
{code}
  public StorageReport[] getStorageReports() {
final StorageReport[] reports = new StorageReport[storageMap.size()];
{code}
Other methods use the following construct:
{code}
synchronized (storageMap) {
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6726) TestNamenodeCapacityReport fails intermittently

2014-07-22 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6726:


 Summary: TestNamenodeCapacityReport fails intermittently
 Key: HDFS-6726
 URL: https://issues.apache.org/jira/browse/HDFS-6726
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From 
https://builds.apache.org/job/Hadoop-Hdfs-trunk/1812/testReport/junit/org.apache.hadoop.hdfs.server.namenode/TestNamenodeCapacityReport/testXceiverCount/
 :
{code}
java.io.IOException: Unable to close file because the last block does not have 
enough number of replicas.
at 
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2141)
at 
org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2109)
at 
org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport.testXceiverCount(TestNamenodeCapacityReport.java:281)
{code}
There were multiple occurrences of 'Broken pipe', 'Connection reset by peer' 
and 'Premature EOF from inputStream' exceptions in test output



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6586) TestBalancer#testExitZeroOnSuccess sometimes fails in trunk

2014-06-22 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6586:


 Summary: TestBalancer#testExitZeroOnSuccess sometimes fails in 
trunk
 Key: HDFS-6586
 URL: https://issues.apache.org/jira/browse/HDFS-6586
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From 
https://builds.apache.org/job/Hadoop-Hdfs-trunk/1782/testReport/org.apache.hadoop.hdfs.server.balancer/TestBalancer/testExitZeroOnSuccess/
 :
{code}
Stacktrace
java.util.concurrent.TimeoutException: Rebalancing expected avg utilization to 
become 0.2, but on datanode 127.0.0.1:49048 it remains at 0.08 after more than 
4 msec.
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.waitForBalancer(TestBalancer.java:284)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.runBalancerCli(TestBalancer.java:392)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:357)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.oneNodeTest(TestBalancer.java:398)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testExitZeroOnSuccess(TestBalancer.java:550)
{code}




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6582) Missing null check in RpcProgramNfs3#read(XDR, SecurityHandler)

2014-06-20 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6582:


 Summary: Missing null check in RpcProgramNfs3#read(XDR, 
SecurityHandler)
 Key: HDFS-6582
 URL: https://issues.apache.org/jira/browse/HDFS-6582
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


Around line 691:
{code}
FSDataInputStream fis = clientCache.getDfsInputStream(userName,
Nfs3Utils.getFileIdPath(handle));

try {
  readCount = fis.read(offset, readbuffer, 0, count);
{code}
fis may be null, leading to NullPointerException



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6501) TestCrcCorruption#testCorruptionDuringWrt sometimes fails in trunk

2014-06-07 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6501:


 Summary: TestCrcCorruption#testCorruptionDuringWrt sometimes fails 
in trunk
 Key: HDFS-6501
 URL: https://issues.apache.org/jira/browse/HDFS-6501
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1767/ :
{code}
REGRESSION:  org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt

Error Message:
test timed out after 5 milliseconds

Stack Trace:
java.lang.Exception: test timed out after 5 milliseconds
at java.lang.Object.wait(Native Method)
at 
org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2024)
at 
org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2008)
at 
org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2107)
at 
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
at 
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:98)
at 
org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt(TestCrcCorruption.java:133)
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6481) DatanodeManager#getDatanodeStorageInfos() should check the length of storageIDs

2014-06-03 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6481:


 Summary: DatanodeManager#getDatanodeStorageInfos() should check 
the length of storageIDs
 Key: HDFS-6481
 URL: https://issues.apache.org/jira/browse/HDFS-6481
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.3.0
Reporter: Ted Yu


Ian Brooks reported the following stack trace:
{code}
2014-06-03 13:05:03,915 WARN  [DataStreamer for file 
/user/hbase/WALs/,16020,1401716790638/%2C16020%2C1401716790638.1401796562200
 block BP-2121456822-10.143.38.149-1396953188241:blk_1074073683_332932] 
hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.lang.ArrayIndexOutOfBoundsException):
 0
at 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getDatanodeStorageInfos(DatanodeManager.java:467)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2779)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:594)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:430)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1958)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1956)

at org.apache.hadoop.ipc.Client.call(Client.java:1347)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy13.getAdditionalDatanode(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolTranslatorPB.java:352)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy14.getAdditionalDatanode(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:266)
at com.sun.proxy.$Proxy15.getAdditionalDatanode(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:919)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:919)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1031)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:823)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:475)
2014-06-03 13:05:48,489 ERROR [RpcServer.handler=22,port=16020] wal.FSHLog: 
syncer encountered error, will retry. txid=211
org.apache.hadoop.ipc.RemoteException(java.lang.ArrayIndexOutOfBoundsException):
 0
at 
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getDatanodeStorageInfos(DatanodeManager.java:467)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:2779)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:594)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:430)
at

[jira] [Created] (HDFS-6437) TestBookKeeperHACheckpoints#TestStandbyCheckpoints fails in trunk

2014-05-20 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6437:


 Summary: TestBookKeeperHACheckpoints#TestStandbyCheckpoints fails 
in trunk
 Key: HDFS-6437
 URL: https://issues.apache.org/jira/browse/HDFS-6437
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


The following test failure can be reproduced locally:
{code}
testSBNCheckpoints(org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints)
  Time elapsed: 2.79 sec   ERROR!
java.lang.NullPointerException: null
at 
org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints.testSBNCheckpoints(TestStandbyCheckpoints.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6415) Missing null check in FSImageSerialization#writePermissionStatus()

2014-05-16 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6415:


 Summary: Missing null check in 
FSImageSerialization#writePermissionStatus()
 Key: HDFS-6415
 URL: https://issues.apache.org/jira/browse/HDFS-6415
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
PermissionStatus.write(out, inode.getUserName(), inode.getGroupName(), p);
{code}
getUserName() / getGroupName() may return null.
null check should be added for these two calls.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6368) TransferFsImage#receiveFile() should perform validation on fsImageName parameter

2014-05-15 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6368:


 Summary: TransferFsImage#receiveFile() should perform validation 
on fsImageName parameter
 Key: HDFS-6368
 URL: https://issues.apache.org/jira/browse/HDFS-6368
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


Currently only null check is performed:
{code}
  if (fsImageName == null) {
throw new IOException(No filename header provided by server);
  }
  newLocalPaths.add(new File(localPath, fsImageName));
{code}
Value of fsImageName, obtained from HttpURLConnection header, may be tainted.
This may allow an attacker to access, modify, or test the existence of critical 
or sensitive files.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6369) RemoteBlockReader#available() should call FSInputChecker.available()

2014-05-11 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6369:


 Summary: RemoteBlockReader#available() should call 
FSInputChecker.available()
 Key: HDFS-6369
 URL: https://issues.apache.org/jira/browse/HDFS-6369
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Trivial


Currently DFSClient.TCP_WINDOW_SIZE is directly returned.
However, FSInputChecker.available(), in the superclass, may return value lower 
than the constant.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HDFS-6083) TestQuorumJournalManager#testChangeWritersLogsOutOfSync2 occasionally fails

2014-05-11 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-6083.
--

Resolution: Cannot Reproduce

 TestQuorumJournalManager#testChangeWritersLogsOutOfSync2 occasionally fails
 ---

 Key: HDFS-6083
 URL: https://issues.apache.org/jira/browse/HDFS-6083
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor

 From 
 https://builds.apache.org/job/Hadoop-Hdfs-trunk/1695/testReport/junit/org.apache.hadoop.hdfs.qjournal.client/TestQuorumJournalManager/testChangeWritersLogsOutOfSync2/
  :
 {code}
 Leaked thread: IPC Client (26533782) connection to /127.0.0.1:57898 from 
 jenkins Id=590 RUNNABLE
  at java.lang.System.arraycopy(Native Method)
  at java.lang.ThreadGroup.remove(ThreadGroup.java:885)
  at java.lang.Thread.exit(Thread.java:672)
 {code}
 The following check should give more time for the threads to shutdown:
 {code}
 // Should not leak clients between tests -- this can cause flaky tests.
 // (See HDFS-4643)
 GenericTestUtils.assertNoThreadsMatching(.*IPC Client.*);
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6290) File is not closed in OfflineImageViewerPB#run()

2014-04-26 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6290:


 Summary: File is not closed in OfflineImageViewerPB#run()
 Key: HDFS-6290
 URL: https://issues.apache.org/jira/browse/HDFS-6290
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
  } else if (processor.equals(XML)) {
new PBImageXmlWriter(conf, out).visit(new RandomAccessFile(inputFile,
r));
{code}
The RandomAccessFile instance should be closed before the method returns.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6291) FSImage may be left unclosed in BootstrapStandby#doRun()

2014-04-26 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6291:


 Summary: FSImage may be left unclosed in BootstrapStandby#doRun()
 Key: HDFS-6291
 URL: https://issues.apache.org/jira/browse/HDFS-6291
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


At around line 203:
{code}
  if (!checkLogsAvailableForRead(image, imageTxId, curTxId)) {
return ERR_CODE_LOGS_UNAVAILABLE;
  }
{code}
If we return following the above check, image is not closed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6264) Provide FileSystem#create() variant which throws exception if parent directory doesn't exist

2014-04-21 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6264:


 Summary: Provide FileSystem#create() variant which throws 
exception if parent directory doesn't exist
 Key: HDFS-6264
 URL: https://issues.apache.org/jira/browse/HDFS-6264
 Project: Hadoop HDFS
  Issue Type: Task
Reporter: Ted Yu
Priority: Minor


FileSystem#createNonRecursive() is deprecated.

However, there is no DistributedFileSystem#create() implementation which throws 
exception if parent directory doesn't exist.
This limits clients' migration away from the deprecated method.

Variant of create() method should be added which throws exception if parent 
directory doesn't exist.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6257) TestCacheDirectives#testExceedsCapacity fails occasionally in trunk

2014-04-18 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6257:


 Summary: TestCacheDirectives#testExceedsCapacity fails 
occasionally in trunk
 Key: HDFS-6257
 URL: https://issues.apache.org/jira/browse/HDFS-6257
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1736/ :

REGRESSION:  
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.testExceedsCapacity
{code}
Error Message:
Namenode should not send extra CACHE commands expected:0 but was:2

Stack Trace:
java.lang.AssertionError: Namenode should not send extra CACHE commands 
expected:0 but was:2
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at 
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives.testExceedsCapacity(TestCacheDirectives.java:1419)
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6177) TestHttpFSServer fails occasionally in trunk

2014-03-30 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6177:


 Summary: TestHttpFSServer fails occasionally in trunk
 Key: HDFS-6177
 URL: https://issues.apache.org/jira/browse/HDFS-6177
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From https://builds.apache.org/job/Hadoop-hdfs-trunk/1716/consoleFull :
{code}
Running org.apache.hadoop.fs.http.server.TestHttpFSServer
Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.424 sec  
FAILURE! - in org.apache.hadoop.fs.http.server.TestHttpFSServer
testDelegationTokenOperations(org.apache.hadoop.fs.http.server.TestHttpFSServer)
  Time elapsed: 0.559 sec   FAILURE!
java.lang.AssertionError: expected:401 but was:403
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.hadoop.fs.http.server.TestHttpFSServer.testDelegationTokenOperations(TestHttpFSServer.java:352)
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6160) TestSafeMode occasionally fails

2014-03-26 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6160:


 Summary: TestSafeMode occasionally fails
 Key: HDFS-6160
 URL: https://issues.apache.org/jira/browse/HDFS-6160
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From 
https://builds.apache.org/job/PreCommit-HDFS-Build/6511//testReport/org.apache.hadoop.hdfs/TestSafeMode/testInitializeReplQueuesEarly/
 :
{code}
java.lang.AssertionError: expected:13 but was:0
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.hadoop.hdfs.TestSafeMode.testInitializeReplQueuesEarly(TestSafeMode.java:212)
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Reopened] (HDFS-5672) TestHASafeMode#testSafeBlockTracking fails in trunk

2014-03-25 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu reopened HDFS-5672:
--


 TestHASafeMode#testSafeBlockTracking fails in trunk
 ---

 Key: HDFS-5672
 URL: https://issues.apache.org/jira/browse/HDFS-5672
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu

 From build #1614:
 {code}
  TestHASafeMode.testSafeBlockTracking:623-assertSafeMode:488 Bad safemode 
 status: 'Safe mode is ON. The reported blocks 3 needs additional 7 blocks to 
 reach the threshold 0.9990 of total blocks 10.
 Safe mode will be turned off automatically'
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6092) DistributedFileSystem#getCanonicalServiceName() and DistributedFileSystem#getUri() may return inconsistent results w.r.t. port

2014-03-11 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6092:


 Summary: DistributedFileSystem#getCanonicalServiceName() and 
DistributedFileSystem#getUri() may return inconsistent results w.r.t. port
 Key: HDFS-6092
 URL: https://issues.apache.org/jira/browse/HDFS-6092
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.3.0
Reporter: Ted Yu


I discovered this when working on HBASE-10717
Here is sample code to reproduce the problem:
{code}
Path desPath = new Path(hdfs://127.0.0.1/);
FileSystem desFs = desPath.getFileSystem(conf);

String s = desFs.getCanonicalServiceName();
URI uri = desFs.getUri();
{code}
Canonical name string contains the default port - 8020
But uri doesn't contain port.
This would result in the following exception:
{code}
testIsSameHdfs(org.apache.hadoop.hbase.util.TestFSHDFSUtils)  Time elapsed: 
0.001 sec   ERROR!
java.lang.IllegalArgumentException: port out of range:-1
at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
at java.net.InetSocketAddress.init(InetSocketAddress.java:224)
at 
org.apache.hadoop.hbase.util.FSHDFSUtils.getNNAddresses(FSHDFSUtils.java:88)
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6081) TestRetryCacheWithHA#testCreateSymlink occasionally fails in trunk

2014-03-09 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6081:


 Summary: TestRetryCacheWithHA#testCreateSymlink occasionally fails 
in trunk
 Key: HDFS-6081
 URL: https://issues.apache.org/jira/browse/HDFS-6081
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu


From 
https://builds.apache.org/job/Hadoop-Hdfs-trunk/1696/testReport/junit/org.apache.hadoop.hdfs.server.namenode.ha/TestRetryCacheWithHA/testCreateSymlink/
 :
{code}
2014-03-09 13:18:47,515 WARN  security.UserGroupInformation 
(UserGroupInformation.java:doAs(1600)) - PriviledgedActionException as:jenkins 
(auth:SIMPLE) cause:java.io.IOException: failed to create link /testlink either 
because the filename is invalid or the file exists
2014-03-09 13:18:47,515 INFO  ipc.Server (Server.java:run(2093)) - IPC Server 
handler 0 on 39303, call 
org.apache.hadoop.hdfs.protocol.ClientProtocol.createSymlink from 
127.0.0.1:32909 Call#682 Retry#1: error: java.io.IOException: failed to create 
link /testlink either because the filename is invalid or the file exists
java.io.IOException: failed to create link /testlink either because the 
filename is invalid or the file exists
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlinkInt(FSNamesystem.java:2053)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlink(FSNamesystem.java:2023)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.createSymlink(NameNodeRpcServer.java:965)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.createSymlink(ClientNamenodeProtocolServerSideTranslatorPB.java:844)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:605)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2071)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2067)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1597)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2065)
2014-03-09 13:18:47,522 INFO  blockmanagement.BlockManager 
(BlockManager.java:processMisReplicatesAsync(2475)) - Total number of blocks
= 1
2014-03-09 13:18:47,523 INFO  blockmanagement.BlockManager 
(BlockManager.java:processMisReplicatesAsync(2476)) - Number of invalid blocks  
= 0
2014-03-09 13:18:47,523 INFO  blockmanagement.BlockManager 
(BlockManager.java:processMisReplicatesAsync(2477)) - Number of 
under-replicated blocks = 0
2014-03-09 13:18:47,523 INFO  ha.TestRetryCacheWithHA 
(TestRetryCacheWithHA.java:run(1162)) - Got Exception while calling 
createSymlink
org.apache.hadoop.ipc.RemoteException(java.io.IOException): failed to create 
link /testlink either because the filename is invalid or the file exists
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlinkInt(FSNamesystem.java:2053)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.createSymlink(FSNamesystem.java:2023)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.createSymlink(NameNodeRpcServer.java:965)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.createSymlink(ClientNamenodeProtocolServerSideTranslatorPB.java:844)
at 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:605)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2071)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2067)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1597)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2065)

at org.apache.hadoop.ipc.Client.call(Client.java:1409)
at org.apache.hadoop.ipc.Client.call(Client.java:1362)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at $Proxy17.createSymlink(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.createSymlink(ClientNamenodeProtocolTranslatorPB.java:794)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at

[jira] [Created] (HDFS-6083) TestQuorumJournalManager#testChangeWritersLogsOutOfSync2 occasionally fails

2014-03-09 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6083:


 Summary: TestQuorumJournalManager#testChangeWritersLogsOutOfSync2 
occasionally fails
 Key: HDFS-6083
 URL: https://issues.apache.org/jira/browse/HDFS-6083
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From 
https://builds.apache.org/job/Hadoop-Hdfs-trunk/1695/testReport/junit/org.apache.hadoop.hdfs.qjournal.client/TestQuorumJournalManager/testChangeWritersLogsOutOfSync2/
 :
{code}
Leaked thread: IPC Client (26533782) connection to /127.0.0.1:57898 from 
jenkins Id=590 RUNNABLE
 at java.lang.System.arraycopy(Native Method)
 at java.lang.ThreadGroup.remove(ThreadGroup.java:885)
 at java.lang.Thread.exit(Thread.java:672)
{code}
The following check should give more time for the threads to shutdown:
{code}
// Should not leak clients between tests -- this can cause flaky tests.
// (See HDFS-4643)
GenericTestUtils.assertNoThreadsMatching(.*IPC Client.*);
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6037) TestIncrementalBlockReports#testReplaceReceivedBlock fails occasionally in trunk

2014-03-01 Thread Ted Yu (JIRA)

Ted Yu created HDFS-6037:


 Summary: TestIncrementalBlockReports#testReplaceReceivedBlock 
fails occasionally in trunk
 Key: HDFS-6037
 URL: https://issues.apache.org/jira/browse/HDFS-6037
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu


From 
https://builds.apache.org/job/Hadoop-Hdfs-trunk/1688/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestIncrementalBlockReports/testReplaceReceivedBlock/
 :
{code}
datanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(
any,
any,
any
);
Wanted 1 time:
- at 
org.apache.hadoop.hdfs.server.datanode.TestIncrementalBlockReports.testReplaceReceivedBlock(TestIncrementalBlockReports.java:198)
But was 2 times. Undesired invocation:
- at 
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.reportReceivedDeletedBlocks(BPServiceActor.java:303)
{code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5997) TestHASafeMode#testBlocksAddedWhileStandbyIsDown fails in trunk

2014-02-22 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5997:


 Summary: TestHASafeMode#testBlocksAddedWhileStandbyIsDown fails in 
trunk
 Key: HDFS-5997
 URL: https://issues.apache.org/jira/browse/HDFS-5997
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu


From https://builds.apache.org/job/Hadoop-Hdfs-trunk/1681/ :

REGRESSION:  
org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode.testBlocksAddedWhileStandbyIsDown

Error Message:
{code}
Bad safemode status: 'Safe mode is ON. The reported blocks 7 has reached the 
threshold 0.9990 of total blocks 6. The number of live datanodes 3 has reached 
the minimum number 0. Safe mode will be turned off automatically in 28 seconds.'
{code}

Stack Trace:
{code}
java.lang.AssertionError: Bad safemode status: 'Safe mode is ON. The reported 
blocks 7 has reached the threshold 0.9990 of total blocks 6. The number of live 
datanodes 3 has reached the minimum number 0. Safe mode will be turned off 
automatically in 28 seconds.'
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode.assertSafeMode(TestHASafeMode.java:493)
at 
org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode.testBlocksAddedWhileStandbyIsDown(TestHASafeMode.java:660)
{code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5953) TestBlockReaderFactory fails in trunk

2014-02-14 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5953:


 Summary: TestBlockReaderFactory fails in trunk
 Key: HDFS-5953
 URL: https://issues.apache.org/jira/browse/HDFS-5953
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu


From 
https://builds.apache.org/job/Hadoop-Hdfs-trunk/1673/testReport/junit/org.apache.hadoop.hdfs/TestBlockReaderFactory/testFallbackFromShortCircuitToUnixDomainTraffic/
 :
{code}
java.lang.RuntimeException: Although a UNIX domain socket path is configured as 
/tmp/socks.1392383436573.1418778351/testFallbackFromShortCircuitToUnixDomainTraffic._PORT,
 we cannot start a localDataXceiverServer because libhadoop cannot be loaded.
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:601)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:573)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:769)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:315)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1864)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1764)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1243)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:699)
at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:359)
at 
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:340)
at 
org.apache.hadoop.hdfs.TestBlockReaderFactory.testFallbackFromShortCircuitToUnixDomainTraffic(TestBlockReaderFactory.java:99)
{code}
This test failure can be reproduced locally (on Mac).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5913) Nfs3Utils#getWccAttr() should check attr parameter against null

2014-02-07 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5913:


 Summary: Nfs3Utils#getWccAttr() should check attr parameter 
against null
 Key: HDFS-5913
 URL: https://issues.apache.org/jira/browse/HDFS-5913
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


In RpcProgramNfs3#commit() :
{code}
  Nfs3FileAttributes postOpAttr = null;
  try {
postOpAttr = writeManager.getFileAttr(dfsClient, handle, iug);
  } catch (IOException e1) {
LOG.info(Can't get postOpAttr for fileId:  + handle.getFileId());
  }
  WccData fileWcc = new WccData(Nfs3Utils.getWccAttr(preOpAttr), 
postOpAttr);
{code}
If there is exception, postOpAttr would be null.
However, Nfs3Utils#getWccAttr() dereferences attr parameter directly.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5897) TestNNWithQJM#testNewNamenodeTakesOverWriter occasionally fails in trunk

2014-02-06 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5897:


 Summary: TestNNWithQJM#testNewNamenodeTakesOverWriter occasionally 
fails in trunk
 Key: HDFS-5897
 URL: https://issues.apache.org/jira/browse/HDFS-5897
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu


From 
https://builds.apache.org/job/Hadoop-Hdfs-trunk/1665/testReport/junit/org.apache.hadoop.hdfs.qjournal/TestNNWithQJM/testNewNamenodeTakesOverWriter/
 :
{code}
java.lang.Exception: test timed out after 3 milliseconds
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
at 
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195)
at 
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
at 
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:412)
at 
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:401)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
{code}
I saw:
{code}
2014-02-06 11:38:37,970 ERROR namenode.EditLogInputStream 
(RedundantEditLogInputStream.java:nextOp(221)) - Got error reading edit log 
input stream 
http://localhost:40509/getJournal?jid=myjournalsegmentTxId=3storageInfo=-51%3A1571339494%3A0%3AtestClusterID;
 failing over to edit log 
http://localhost:56244/getJournal?jid=myjournalsegmentTxId=3storageInfo=-51%3A1571339494%3A0%3AtestClusterID
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException:
 got premature end-of-file at txid 0; expected file to go up to 4
at 
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
at 
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:83)
at 
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:140)
at 
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at 
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:83)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:167)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:120)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:708)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:606)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:263)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:874)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:634)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:446)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:502)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:658)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:643)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1291)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:939)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:824)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:678)
at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:359)
at 
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:340)
at 
org.apache.hadoop.hdfs.qjournal.TestNNWithQJM.testNewNamenodeTakesOverWriter(TestNNWithQJM.java:145)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
{code}



--

[jira] [Created] (HDFS-5886) Potential null pointer deference in RpcProgramNfs3#readlink()

2014-02-04 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5886:


 Summary: Potential null pointer deference in 
RpcProgramNfs3#readlink()
 Key: HDFS-5886
 URL: https://issues.apache.org/jira/browse/HDFS-5886
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu


Here is related code:
{code}
  if (MAX_READ_TRANSFER_SIZE  target.getBytes().length) {
return new READLINK3Response(Nfs3Status.NFS3ERR_IO, postOpAttr, null);
  }
{code}
READLINK3Response constructor would dereference the third parameter:
{code}
this.path = new byte[path.length];
{code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5834) TestCheckpoint#testCheckpoint may fail due to Bad value assertion

2014-01-26 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5834:


 Summary: TestCheckpoint#testCheckpoint may fail due to Bad value 
assertion
 Key: HDFS-5834
 URL: https://issues.apache.org/jira/browse/HDFS-5834
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


I saw the following when running test suite on Linux:
{code}
testCheckpoint(org.apache.hadoop.hdfs.server.namenode.TestCheckpoint)  Time 
elapsed: 3.058 sec   FAILURE!
java.lang.AssertionError: Bad value for metric GetImageNumOps
Expected: gt(0)
 got: 0L

at org.junit.Assert.assertThat(Assert.java:780)
at 
org.apache.hadoop.test.MetricsAsserts.assertCounterGt(MetricsAsserts.java:318)
at 
org.apache.hadoop.hdfs.server.namenode.TestCheckpoint.testCheckpoint(TestCheckpoint.java:1058)
{code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5750) JHLogAnalyzer#parseLogFile() should close stm upon return

2014-01-09 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5750:


 Summary: JHLogAnalyzer#parseLogFile() should close stm upon return
 Key: HDFS-5750
 URL: https://issues.apache.org/jira/browse/HDFS-5750
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


stm is assigned to in
But stm may point to another InputStream :
{code}
if(compressionClass != null) {
  CompressionCodec codec = (CompressionCodec)
ReflectionUtils.newInstance(compressionClass, new Configuration());
  in = codec.createInputStream(stm);
{code}
stm should be closed in the finally block.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5721) sharedEditsImage in Namenode#initializeSharedEdits() should be closed before method returns

2014-01-06 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5721:


 Summary: sharedEditsImage in Namenode#initializeSharedEdits() 
should be closed before method returns
 Key: HDFS-5721
 URL: https://issues.apache.org/jira/browse/HDFS-5721
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


At line 901:
{code}
  FSImage sharedEditsImage = new FSImage(conf,
  Lists.URInewArrayList(),
  sharedEditsDirs);
{code}
sharedEditsImage is not closed before the method returns.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5719) FSImage#doRollback() should close prevState before return

2014-01-05 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5719:


 Summary: FSImage#doRollback() should close prevState before return
 Key: HDFS-5719
 URL: https://issues.apache.org/jira/browse/HDFS-5719
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Priority: Minor


{code}
FSImage prevState = new FSImage(conf);
{code}
prevState should be closed before return from doRollback()



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5718) TestHttpsFileSystem intermittently fails with Port in use error

2014-01-04 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5718:


 Summary: TestHttpsFileSystem intermittently fails with Port in use 
error
 Key: HDFS-5718
 URL: https://issues.apache.org/jira/browse/HDFS-5718
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu
Priority: Minor


From 
https://builds.apache.org/job/Hadoop-Hdfs-trunk/1634/testReport/junit/org.apache.hadoop.hdfs.web/TestHttpsFileSystem/org_apache_hadoop_hdfs_web_TestHttpsFileSystem/
 :
{code}
java.net.BindException: Port in use: localhost:50475
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383)
at java.net.ServerSocket.bind(ServerSocket.java:328)
at java.net.ServerSocket.init(ServerSocket.java:194)
at javax.net.ssl.SSLServerSocket.init(SSLServerSocket.java:106)
at 
com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.init(SSLServerSocketImpl.java:108)
at 
com.sun.net.ssl.internal.ssl.SSLServerSocketFactoryImpl.createServerSocket(SSLServerSocketFactoryImpl.java:72)
at 
org.mortbay.jetty.security.SslSocketConnector.newServerSocket(SslSocketConnector.java:478)
at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)
at org.apache.hadoop.http.HttpServer.openListeners(HttpServer.java:973)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:914)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:412)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:769)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:315)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1846)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1746)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1203)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:673)
at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:342)
at 
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:323)
at 
org.apache.hadoop.hdfs.web.TestHttpsFileSystem.setUp(TestHttpsFileSystem.java:64)
{code}
This could have been caused by concurrent test(s).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5705) TestSecondaryNameNodeUpgrade#testChangeNsIDFails may fail due to ConcurrentModificationException

2013-12-29 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5705:


 Summary: TestSecondaryNameNodeUpgrade#testChangeNsIDFails may fail 
due to ConcurrentModificationException
 Key: HDFS-5705
 URL: https://issues.apache.org/jira/browse/HDFS-5705
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu


From 
https://builds.apache.org/job/Hadoop-Hdfs-trunk/1626/testReport/org.apache.hadoop.hdfs.server.namenode/TestSecondaryNameNodeUpgrade/testChangeNsIDFails/
 :
{code}
java.util.ConcurrentModificationException: null
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
at java.util.HashMap$EntryIterator.next(HashMap.java:834)
at java.util.HashMap$EntryIterator.next(HashMap.java:832)
at 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.shutdown(FsVolumeImpl.java:251)
at 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.shutdown(FsVolumeList.java:218)
at 
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.shutdown(FsDatasetImpl.java:1414)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:1309)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdownDataNodes(MiniDFSCluster.java:1464)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1439)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1423)
at 
org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade.doIt(TestSecondaryNameNodeUpgrade.java:97)
at 
org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade.testChangeNsIDFails(TestSecondaryNameNodeUpgrade.java:116)
{code}
The above happens when shutdown() is called in parallel to addBlockPool() or 
shutdownBlockPool().



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Resolved] (HDFS-5672) TestHASafeMode#testSafeBlockTracking fails in trunk

2013-12-26 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-5672.
--

Resolution: Cannot Reproduce

 TestHASafeMode#testSafeBlockTracking fails in trunk
 ---

 Key: HDFS-5672
 URL: https://issues.apache.org/jira/browse/HDFS-5672
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu

 From build #1614:
 {code}
  TestHASafeMode.testSafeBlockTracking:623-assertSafeMode:488 Bad safemode 
 status: 'Safe mode is ON. The reported blocks 3 needs additional 7 blocks to 
 reach the threshold 0.9990 of total blocks 10.
 Safe mode will be turned off automatically'
 {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-5679) TestCacheDirectives should handle the case where native code is not available

2013-12-18 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5679:


 Summary: TestCacheDirectives should handle the case where native 
code is not available
 Key: HDFS-5679
 URL: https://issues.apache.org/jira/browse/HDFS-5679
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu


TestCacheDirectives fails on trunk due to:
{code}
testBasicPoolOperations(org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives)
  Time elapsed: 1.618 sec   ERROR!
java.lang.RuntimeException: Cannot start datanode because the configured max 
locked memory size (dfs.datanode.max.locked.memory) is greater than zero and 
native code is not available.
{code}
Configuration of max locked memory size should be dependent on whether native 
code is available.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HDFS-5672) TestHASafeMode#testSafeBlockTracking fails in trunk

2013-12-16 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5672:


 Summary: TestHASafeMode#testSafeBlockTracking fails in trunk
 Key: HDFS-5672
 URL: https://issues.apache.org/jira/browse/HDFS-5672
 Project: Hadoop HDFS
  Issue Type: Test
Reporter: Ted Yu


From build #1614:
{code}
 TestHASafeMode.testSafeBlockTracking:623-assertSafeMode:488 Bad safemode 
status: 'Safe mode is ON. The reported blocks 3 needs additional 7 blocks to 
reach the threshold 0.9990 of total blocks 10.
Safe mode will be turned off automatically'
{code}



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

[jira] [Created] (HDFS-5576) RPC#stopProxy() should log the class of proxy when IllegalArgumentException is encountered

2013-11-27 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5576:


 Summary: RPC#stopProxy() should log the class of proxy when 
IllegalArgumentException is encountered
 Key: HDFS-5576
 URL: https://issues.apache.org/jira/browse/HDFS-5576
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Ted Yu
Priority: Minor


When investigating HBASE-10029, [~szetszwo] made the suggestion of logging the 
class of proxy when IllegalArgumentException is thrown.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (HDFS-4642) Allow lease recovery for multiple paths to be issued in one request

2013-11-11 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/HDFS-4642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HDFS-4642.
--

Resolution: Later

 Allow lease recovery for multiple paths to be issued in one request
 ---

 Key: HDFS-4642
 URL: https://issues.apache.org/jira/browse/HDFS-4642
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Ted Yu

 Currently client can only request lease recovery for one Path:
 {code}
   public boolean recoverLease(Path f) throws IOException {
 {code}
 For HBase distributed log splitting, Nicolas made a suggestion here:
 https://issues.apache.org/jira/browse/HBASE-7878?focusedCommentId=13615364page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13615364
 HBase master collects the files that should be split, it issues lease 
 recovery for the files (in one request), then distribute log splitting.
 This would help shorten MTTR.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HDFS-5352) Server#initLog() doesn't close InputStream

2013-10-11 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5352:


 Summary: Server#initLog() doesn't close InputStream
 Key: HDFS-5352
 URL: https://issues.apache.org/jira/browse/HDFS-5352
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
 Attachments: hdfs-5352.patch

Here is related code snippet in 
hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/server/Server.java:
{code}
  Properties props = new Properties();
  try {
InputStream is = getResource(DEFAULT_LOG4J_PROPERTIES);
props.load(is);
  } catch (IOException ex) {
throw new ServerException(ServerException.ERROR.S03, 
DEFAULT_LOG4J_PROPERTIES, ex.getMessage(), ex);
  }
{code}
is should be closed after loading.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (HDFS-5081) DistributedFileSystem#listStatus() throws FileNotFoundException when target of symlink doesn't exist

2013-08-08 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5081:


 Summary: DistributedFileSystem#listStatus() throws 
FileNotFoundException when target of symlink doesn't exist
 Key: HDFS-5081
 URL: https://issues.apache.org/jira/browse/HDFS-5081
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu


I was running HBase trunk test suite against hadoop 2.1.1-SNAPSHOT.
One test failed due to:
{code}
org.apache.hadoop.hbase.catalog.TestMetaMigrationConvertingToPB  Time elapsed: 
1,594,938.629 sec   ERROR!
java.io.FileNotFoundException: File 
hdfs://localhost:61300/user/tyu/hbase/.archive does not exist.
  at 
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:656)
  at 
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:92)
  at 
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:714)
  at 
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:710)
  at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:78)
  at 
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:710)
  at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1478)
  at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1518)
  at org.apache.hadoop.hbase.util.FSUtils.getLocalTableDirs(FSUtils.java:1317)
  at 
org.apache.hadoop.hbase.migration.NamespaceUpgrade.migrateTables(NamespaceUpgrade.java:114)
  at 
org.apache.hadoop.hbase.migration.NamespaceUpgrade.upgradeTableDirs(NamespaceUpgrade.java:87)
  at 
org.apache.hadoop.hbase.migration.NamespaceUpgrade.run(NamespaceUpgrade.java:206)
{code}
TestMetaMigrationConvertToPB.tgz was generated from previous release of HBase.
TestMetaMigrationConvertToPB would upgrade to current release of HBase.

The test is at 
hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaMigrationConvertingToPB.java
 under HBase trunk.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-5041) Add the time of last heartbeat to dead server Web UI

2013-07-29 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5041:


 Summary: Add the time of last heartbeat to dead server Web UI
 Key: HDFS-5041
 URL: https://issues.apache.org/jira/browse/HDFS-5041
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Ted Yu
Priority: Minor


In Live Server page, there is a column 'Last Contact'.

On the dead server page, similar column can be added which shows when the last 
heartbeat came from the respective dead node.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-5018) Misspelled DFSConfigKeys#DFS_NAMENODE_STALE_DATANODE_INTERVAL_DEFAULT in javadoc of DatanodeInfo#isStale()

2013-07-21 Thread Ted Yu (JIRA)

Ted Yu created HDFS-5018:


 Summary: Misspelled 
DFSConfigKeys#DFS_NAMENODE_STALE_DATANODE_INTERVAL_DEFAULT in javadoc of 
DatanodeInfo#isStale()
 Key: HDFS-5018
 URL: https://issues.apache.org/jira/browse/HDFS-5018
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Ted Yu


DFSConfigKeys#DFS_NAMENODE_STALE_DATANODE_INTERVAL_DEFAULT was misspelled in 
javadoc of DatanodeInfo#isStale()

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

1 2 >

1 - 100 of 108 matches

Mail list logo