[jira] [Commented] (HDFS-3385) ClassCastException when trying to append a file
[
https://issues.apache.org/jira/browse/HDFS-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271231#comment-13271231
]
Brahma Reddy Battula commented on HDFS-3385:
Hi Nichalos,,thanks for look..
Actually I didn't get exact scenario to reproduce.
First, let me try to reproduce this issue and then I'll verify your patch.
*fulltrace when I got*
{noformat}
java.lang.ClassCastException:
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo cannot be cast to
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoUnderConstruction
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1782)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1580)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1819)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:416)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:217)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42592)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:417)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:891)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1661)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1657)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1205)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1655)
at org.apache.hadoop.ipc.Client.call(Client.java:1159)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:185)
at $Proxy9.append(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:165)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:84)
at $Proxy9.append(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.append(ClientNamenodeProtocolTranslatorPB.java:214)
at org.apache.hadoop.hdfs.DFSClient.callAppend(DFSClient.java:1178)
at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1217)
at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1205)
at
org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:209)
at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:973)
at
com.huawei.isap.test.smartump.hdfs.TestHDFSAPI.appendFile(TestHDFSAPI.java:76)
at
com.huawei.isap.test.smartump.hdfs.TestHDFSAPI.main(TestHDFSAPI.java:40
{noformat}
> ClassCastException when trying to append a file
> ---
>
> Key: HDFS-3385
> URL: https://issues.apache.org/jira/browse/HDFS-3385
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 3.0.0
> Environment: HDFS
>Reporter: Brahma Reddy Battula
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: h3385_20120508.patch
>
>
> When I try to append a file I got
> {noformat}
> 2012-05-08 18:13:40,506 WARN util.KerberosName
> (KerberosName.java:(87)) - Kerberos krb5 configuration not found,
> setting default realm to empty
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo cannot be cast to
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoUnderConstruction
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1787)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1584)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1824)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:425)
> ...
> at org.apache.hadoop.hdfs.DFSClient.callAppend(DFSClient.java:1150)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1189)
> at org.apache.hadoop.hdfs.DFSCli
[jira] [Commented] (HDFS-3385) ClassCastException when trying to append a file
[
https://issues.apache.org/jira/browse/HDFS-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271258#comment-13271258
]
Brahma Reddy Battula commented on HDFS-3385:
Hi Nichalos..
I have reproduced same..
Scenario:
write a file
append a file so that it fails and close the clients
again try to append the same file we will get ClassCastException.
I verified your patch..it's fine..:)
> ClassCastException when trying to append a file
> ---
>
> Key: HDFS-3385
> URL: https://issues.apache.org/jira/browse/HDFS-3385
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 3.0.0
> Environment: HDFS
>Reporter: Brahma Reddy Battula
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: h3385_20120508.patch
>
>
> When I try to append a file I got
> {noformat}
> 2012-05-08 18:13:40,506 WARN util.KerberosName
> (KerberosName.java:(87)) - Kerberos krb5 configuration not found,
> setting default realm to empty
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo cannot be cast to
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoUnderConstruction
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1787)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1584)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1824)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:425)
> ...
> at org.apache.hadoop.hdfs.DFSClient.callAppend(DFSClient.java:1150)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1189)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1177)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:221)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:1)
> at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:981)
> at
> org.apache.hadoop.hdfs.server.datanode.DeleteMe.main(DeleteMe.java:26)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-234) Integration with BookKeeper logging system
[ https://issues.apache.org/jira/browse/HDFS-234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reopened HDFS-234: -- > Integration with BookKeeper logging system > -- > > Key: HDFS-234 > URL: https://issues.apache.org/jira/browse/HDFS-234 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Luca Telloli >Assignee: Ivan Kelly > Fix For: 3.0.0 > > Attachments: HADOOP-5189-trunk-preview.patch, > HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, > HADOOP-5189-v.19.patch, HADOOP-5189.patch, HDFS-234-branch-2.patch, > HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, > HDFS-234.patch, create.png, hdfs_tpt_lat.pdf, zookeeper-dev-bookkeeper.jar, > zookeeper-dev.jar > > > BookKeeper is a system to reliably log streams of records > (https://issues.apache.org/jira/browse/ZOOKEEPER-276). The NameNode is a > natural target for such a system for being the metadata repository of the > entire file system for HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-234) Integration with BookKeeper logging system
[ https://issues.apache.org/jira/browse/HDFS-234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-234: - Fix Version/s: 2.0.0 Hadoop Flags: Reviewed > Integration with BookKeeper logging system > -- > > Key: HDFS-234 > URL: https://issues.apache.org/jira/browse/HDFS-234 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Luca Telloli >Assignee: Ivan Kelly > Fix For: 2.0.0, 3.0.0 > > Attachments: HADOOP-5189-trunk-preview.patch, > HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, > HADOOP-5189-v.19.patch, HADOOP-5189.patch, HDFS-234-branch-2.patch, > HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, > HDFS-234.patch, create.png, hdfs_tpt_lat.pdf, zookeeper-dev-bookkeeper.jar, > zookeeper-dev.jar > > > BookKeeper is a system to reliably log streams of records > (https://issues.apache.org/jira/browse/ZOOKEEPER-276). The NameNode is a > natural target for such a system for being the metadata repository of the > entire file system for HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-234) Integration with BookKeeper logging system
[ https://issues.apache.org/jira/browse/HDFS-234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-234. -- Resolution: Fixed I have just pushed this to Hadoop-2 and alpha cut. Thanks a lot Ivan. > Integration with BookKeeper logging system > -- > > Key: HDFS-234 > URL: https://issues.apache.org/jira/browse/HDFS-234 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Luca Telloli >Assignee: Ivan Kelly > Fix For: 2.0.0, 3.0.0 > > Attachments: HADOOP-5189-trunk-preview.patch, > HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, > HADOOP-5189-v.19.patch, HADOOP-5189.patch, HDFS-234-branch-2.patch, > HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, > HDFS-234.patch, create.png, hdfs_tpt_lat.pdf, zookeeper-dev-bookkeeper.jar, > zookeeper-dev.jar > > > BookKeeper is a system to reliably log streams of records > (https://issues.apache.org/jira/browse/ZOOKEEPER-276). The NameNode is a > natural target for such a system for being the metadata repository of the > entire file system for HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3384) DataStreamer thread should be closed immediatly when failed to setup a PipelineForAppendOrRecovery
[
https://issues.apache.org/jira/browse/HDFS-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271372#comment-13271372
]
amith commented on HDFS-3384:
-
Yes I will provide the patch prepared by EOD
> DataStreamer thread should be closed immediatly when failed to setup a
> PipelineForAppendOrRecovery
> --
>
> Key: HDFS-3384
> URL: https://issues.apache.org/jira/browse/HDFS-3384
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 2.0.0
>Reporter: Brahma Reddy Battula
>Assignee: amith
>
> Scenraio:
> =
> write a file
> corrupt block manually
> call append..
> {noformat}
> 2012-04-19 09:33:10,776 INFO hdfs.DFSClient
> (DFSOutputStream.java:createBlockOutputStream(1059)) - Exception in
> createBlockOutputStream
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1039)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:939)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> 2012-04-19 09:33:10,807 WARN hdfs.DFSClient (DFSOutputStream.java:run(549))
> - DataStreamer Exception
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:510)
> 2012-04-19 09:33:10,807 WARN hdfs.DFSClient
> (DFSOutputStream.java:hflush(1511)) - Error while syncing
> java.io.IOException: All datanodes 10.18.40.20:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:908)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> java.io.IOException: All datanodes 10.18.40.20:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:908)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3384) DataStreamer thread should be closed immediatly when failed to setup a PipelineForAppendOrRecovery
[
https://issues.apache.org/jira/browse/HDFS-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G reassigned HDFS-3384:
-
Assignee: amith (was: Uma Maheswara Rao G)
Amith, could please update the patch?
> DataStreamer thread should be closed immediatly when failed to setup a
> PipelineForAppendOrRecovery
> --
>
> Key: HDFS-3384
> URL: https://issues.apache.org/jira/browse/HDFS-3384
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 2.0.0
>Reporter: Brahma Reddy Battula
>Assignee: amith
>
> Scenraio:
> =
> write a file
> corrupt block manually
> call append..
> {noformat}
> 2012-04-19 09:33:10,776 INFO hdfs.DFSClient
> (DFSOutputStream.java:createBlockOutputStream(1059)) - Exception in
> createBlockOutputStream
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1039)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:939)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> 2012-04-19 09:33:10,807 WARN hdfs.DFSClient (DFSOutputStream.java:run(549))
> - DataStreamer Exception
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:510)
> 2012-04-19 09:33:10,807 WARN hdfs.DFSClient
> (DFSOutputStream.java:hflush(1511)) - Error while syncing
> java.io.IOException: All datanodes 10.18.40.20:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:908)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> java.io.IOException: All datanodes 10.18.40.20:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:908)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3157) Error in deleting block is keep on coming from DN even after the block report and directory scanning has happened
[
https://issues.apache.org/jira/browse/HDFS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271383#comment-13271383
]
Hudson commented on HDFS-3157:
--
Integrated in Hadoop-Hdfs-trunk #1039 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1039/])
HDFS-3157. Error in deleting block is keep on coming from DN even after the
block report and directory scanning has happened. Contributed by Ashish Singhi.
(Revision 1335719)
Result = FAILURE
umamahesh :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335719
Files :
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRBWBlockInvalidation.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java
> Error in deleting block is keep on coming from DN even after the block report
> and directory scanning has happened
> -
>
> Key: HDFS-3157
> URL: https://issues.apache.org/jira/browse/HDFS-3157
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: J.Andreina
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HDFS-3157.patch, HDFS-3157.patch, HDFS-3157.patch
>
>
> Cluster setup:
> 1NN,Three DN(DN1,DN2,DN3),replication factor-2,"dfs.blockreport.intervalMsec"
> 300,"dfs.datanode.directoryscan.interval" 1
> step 1: write one file "a.txt" with sync(not closed)
> step 2: Delete the blocks in one of the datanode say DN1(from rbw) to which
> replication happened.
> step 3: close the file.
> Since the replication factor is 2 the blocks are replicated to the other
> datanode.
> Then at the NN side the following cmd is issued to DN from which the block is
> deleted
> -
> {noformat}
> 2012-03-19 13:41:36,905 INFO org.apache.hadoop.hdfs.StateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: duplicate requested for
> blk_2903555284838653156 to add as corrupt on XX.XX.XX.XX by /XX.XX.XX.XX
> because reported RBW replica with genstamp 1002 does not match COMPLETE
> block's genstamp in block map 1003
> 2012-03-19 13:41:39,588 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> Removing block blk_2903555284838653156_1003 from neededReplications as it has
> enough replicas.
> {noformat}
> From the datanode side in which the block is deleted the following exception
> occured
> {noformat}
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unexpected error trying to delete block blk_2903555284838653156_1003.
> BlockInfo not found in volumeMap.
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:2061)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:581)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:545)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:690)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:522)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:662)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3363) blockmanagement should stop using INodeFile & INodeFileUC
[ https://issues.apache.org/jira/browse/HDFS-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271387#comment-13271387 ] Hudson commented on HDFS-3363: -- Integrated in Hadoop-Hdfs-trunk #1039 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1039/]) Remove the empty file FSInodeInfo.java for HDFS-3363. (Revision 1335788) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335788 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSInodeInfo.java > blockmanagement should stop using INodeFile & INodeFileUC > -- > > Key: HDFS-3363 > URL: https://issues.apache.org/jira/browse/HDFS-3363 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 2.0.0, 3.0.0 >Reporter: John George >Assignee: John George >Priority: Minor > Fix For: 2.0.0 > > Attachments: HDFS-3363.java, HDFS-3363.java, HDFS-3363.java, > HDFS-3363.patch > > > Blockmanagement should stop using INodeFile and INodeFileUnderConstruction. > One way would be to create an interface, like BlockColletion, that is passed > along to the blockmanagement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3157) Error in deleting block is keep on coming from DN even after the block report and directory scanning has happened
[
https://issues.apache.org/jira/browse/HDFS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271425#comment-13271425
]
Hudson commented on HDFS-3157:
--
Integrated in Hadoop-Mapreduce-trunk #1074 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1074/])
HDFS-3157. Error in deleting block is keep on coming from DN even after the
block report and directory scanning has happened. Contributed by Ashish Singhi.
(Revision 1335719)
Result = SUCCESS
umamahesh :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335719
Files :
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestRBWBlockInvalidation.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java
> Error in deleting block is keep on coming from DN even after the block report
> and directory scanning has happened
> -
>
> Key: HDFS-3157
> URL: https://issues.apache.org/jira/browse/HDFS-3157
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: J.Andreina
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HDFS-3157.patch, HDFS-3157.patch, HDFS-3157.patch
>
>
> Cluster setup:
> 1NN,Three DN(DN1,DN2,DN3),replication factor-2,"dfs.blockreport.intervalMsec"
> 300,"dfs.datanode.directoryscan.interval" 1
> step 1: write one file "a.txt" with sync(not closed)
> step 2: Delete the blocks in one of the datanode say DN1(from rbw) to which
> replication happened.
> step 3: close the file.
> Since the replication factor is 2 the blocks are replicated to the other
> datanode.
> Then at the NN side the following cmd is issued to DN from which the block is
> deleted
> -
> {noformat}
> 2012-03-19 13:41:36,905 INFO org.apache.hadoop.hdfs.StateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: duplicate requested for
> blk_2903555284838653156 to add as corrupt on XX.XX.XX.XX by /XX.XX.XX.XX
> because reported RBW replica with genstamp 1002 does not match COMPLETE
> block's genstamp in block map 1003
> 2012-03-19 13:41:39,588 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> Removing block blk_2903555284838653156_1003 from neededReplications as it has
> enough replicas.
> {noformat}
> From the datanode side in which the block is deleted the following exception
> occured
> {noformat}
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unexpected error trying to delete block blk_2903555284838653156_1003.
> BlockInfo not found in volumeMap.
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:2061)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:581)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:545)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:690)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:522)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:662)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3363) blockmanagement should stop using INodeFile & INodeFileUC
[ https://issues.apache.org/jira/browse/HDFS-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271429#comment-13271429 ] Hudson commented on HDFS-3363: -- Integrated in Hadoop-Mapreduce-trunk #1074 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1074/]) Remove the empty file FSInodeInfo.java for HDFS-3363. (Revision 1335788) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1335788 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSInodeInfo.java > blockmanagement should stop using INodeFile & INodeFileUC > -- > > Key: HDFS-3363 > URL: https://issues.apache.org/jira/browse/HDFS-3363 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 2.0.0, 3.0.0 >Reporter: John George >Assignee: John George >Priority: Minor > Fix For: 2.0.0 > > Attachments: HDFS-3363.java, HDFS-3363.java, HDFS-3363.java, > HDFS-3363.patch > > > Blockmanagement should stop using INodeFile and INodeFileUnderConstruction. > One way would be to create an interface, like BlockColletion, that is passed > along to the blockmanagement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3391) Failing tests in branch-2
[ https://issues.apache.org/jira/browse/HDFS-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George reassigned HDFS-3391: - Assignee: John George > Failing tests in branch-2 > - > > Key: HDFS-3391 > URL: https://issues.apache.org/jira/browse/HDFS-3391 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Arun C Murthy >Assignee: John George >Priority: Critical > Fix For: 2.0.0 > > > Running org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.208 sec <<< > FAILURE! > -- > Running org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover > Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 81.195 sec > <<< FAILURE! > -- -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3391) Failing tests in branch-2
[ https://issues.apache.org/jira/browse/HDFS-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271434#comment-13271434 ] John George commented on HDFS-3391: --- TestPipelinesFailover passes for me consistently on trunk - ran 5 times. Checking branch-2.0.0-alpha. Ideally failures related to HDFS-3363 would be more consistent since it was mostly name change and package location change. > Failing tests in branch-2 > - > > Key: HDFS-3391 > URL: https://issues.apache.org/jira/browse/HDFS-3391 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Arun C Murthy >Priority: Critical > Fix For: 2.0.0 > > > Running org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.208 sec <<< > FAILURE! > -- > Running org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover > Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 81.195 sec > <<< FAILURE! > -- -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3370) HDFS hardlink
[ https://issues.apache.org/jira/browse/HDFS-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271450#comment-13271450 ] M. C. Srivas commented on HDFS-3370: Creating hard-links in a distributed file-system will cause all kinds of future problems with scalability. Hard-links are rarely used in the real-world, because of all the associated bizzare problems. Eg, consider a hardlink setup as follows: link1: /path1/dirA/file link2: /path2/dirB/file 1. Changing the permissions along the path "/path1/dirA" to make "file" inaccessible works, but now with hard-links "/path2/dirB" is wide open. 2. Rename "/path2/dirB" to "/path3/dirC" will require taking locks on "/path1/dirA" ... but the "file" does not have "parent ptrs" to figure out which path(s) to lock. I would recommend finding a different approach to implementing snapshots than adding this feature. > HDFS hardlink > - > > Key: HDFS-3370 > URL: https://issues.apache.org/jira/browse/HDFS-3370 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Hairong Kuang >Assignee: Liyin Tang > Attachments: HDFS-HardLink.pdf > > > We'd like to add a new feature hardlink to HDFS that allows harlinked files > to share data without copying. Currently we will support hardlinking only > closed files, but it could be extended to unclosed files as well. > Among many potential use cases of the feature, the following two are > primarily used in facebook: > 1. This provides a lightweight way for applications like hbase to create a > snapshot; > 2. This also allows an application like Hive to move a table to a different > directory without breaking current running hive queries. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3358) Specify explicitly that the NN UI status total is talking of persistent objects on heap.
[
https://issues.apache.org/jira/browse/HDFS-3358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271454#comment-13271454
]
Harsh J commented on HDFS-3358:
---
bq. Should we include symlinks?
Instead of making it "X files, directories and symlinks" and given that
symlinks are a purely optional usage, I'd prefer if we rather just printed "X
inodes"
bq. Should we separate the counts for files, directories and symlinks?
This would be lovely to have instead of inode counts. Is there such metrics
already available?
bq. Should we handle singular/plural? "1 blocks" does not look good.
Yes we should handle this. I'll update this logic in the patch I'll put up next.
bq Are there similar changes on the web UI?
Not sure I understood this one.
bq. We should also refactor the code, so that the same string is not
constructed in two different places.
Agreed and will do.
Do lemme know thoughts on (1) and possibility of (2)!
> Specify explicitly that the NN UI status total is talking of persistent
> objects on heap.
>
>
> Key: HDFS-3358
> URL: https://issues.apache.org/jira/browse/HDFS-3358
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
>Affects Versions: 2.0.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Labels: nitpick
> Attachments: HDFS-3358.patch, HDFS-3358.patch
>
>
> The NN shows, on its web UI, something like:
> {{223 files and directories, 138 blocks = 361 total.}}
> Followed by heap stats.
> We should clarify this line is talking of objects and is related to the heap
> summaries. Perhaps just being explicit about java-terms would be nicer:
> {{223 files and directories, 138 blocks = 361 total objects.}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3370) HDFS hardlink
[ https://issues.apache.org/jira/browse/HDFS-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271453#comment-13271453 ] Daryn Sharp commented on HDFS-3370: --- I fully agree that posix and/or linux conventions should ideally be followed. I did a little testing and didn't realize that a hard link retains the same attrs (owner, group, perms, xattrs, etc) as the original file. Changing one implicit changes the others, so that negates some issues such as differing replication factor concerns. Perhaps hard link creation can be restricted to only the file owner and superuser. The quota concerns are still a bit more complex. Unixy systems like linux and bsd only have fs level quotas for users, so quota handling is trivial compared to directory level quotas in hdfs. Since all hard links implicitly have the same owner, quotas are as simple as incrementing the user's ds quota is at file creation, and decrement when all links are removed. This why a DOS is possible against a user. I'm sorry if I'm missing a detail, but I remain unclear on how you are proposing to handle the directory level quotas. I don't fully grok how finding a common ancestor with a quota is sufficient because quotas can be added or removed at any time. Maybe part of the issue too is I have nested directories with individual quotas in mind, whereas maybe you are assuming one and only one quota from the root? I look forward to your thoughts. > HDFS hardlink > - > > Key: HDFS-3370 > URL: https://issues.apache.org/jira/browse/HDFS-3370 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Hairong Kuang >Assignee: Liyin Tang > Attachments: HDFS-HardLink.pdf > > > We'd like to add a new feature hardlink to HDFS that allows harlinked files > to share data without copying. Currently we will support hardlinking only > closed files, but it could be extended to unclosed files as well. > Among many potential use cases of the feature, the following two are > primarily used in facebook: > 1. This provides a lightweight way for applications like hbase to create a > snapshot; > 2. This also allows an application like Hive to move a table to a different > directory without breaking current running hive queries. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3358) Specify explicitly that the NN UI status total is talking of persistent objects on heap.
[
https://issues.apache.org/jira/browse/HDFS-3358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated HDFS-3358:
--
Status: Open (was: Patch Available)
> Specify explicitly that the NN UI status total is talking of persistent
> objects on heap.
>
>
> Key: HDFS-3358
> URL: https://issues.apache.org/jira/browse/HDFS-3358
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
>Affects Versions: 2.0.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Labels: nitpick
> Attachments: HDFS-3358.patch, HDFS-3358.patch
>
>
> The NN shows, on its web UI, something like:
> {{223 files and directories, 138 blocks = 361 total.}}
> Followed by heap stats.
> We should clarify this line is talking of objects and is related to the heap
> summaries. Perhaps just being explicit about java-terms would be nicer:
> {{223 files and directories, 138 blocks = 361 total objects.}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3391) Failing tests in branch-2
[ https://issues.apache.org/jira/browse/HDFS-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3391: - Target Version/s: 2.0.0 Fix Version/s: (was: 2.0.0) > Failing tests in branch-2 > - > > Key: HDFS-3391 > URL: https://issues.apache.org/jira/browse/HDFS-3391 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Arun C Murthy >Assignee: John George >Priority: Critical > > Running org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.208 sec <<< > FAILURE! > -- > Running org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover > Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 81.195 sec > <<< FAILURE! > -- -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3328) NPE in DataNode.getIpcPort
[
https://issues.apache.org/jira/browse/HDFS-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271496#comment-13271496
]
Aaron T. Myers commented on HDFS-3328:
--
+1, the patch looks good to me.
> NPE in DataNode.getIpcPort
> --
>
> Key: HDFS-3328
> URL: https://issues.apache.org/jira/browse/HDFS-3328
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 2.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Eli Collins
>Priority: Minor
> Attachments: hdfs-3328.txt
>
>
> While running the tests, I have seen this exceptions.Tests passed.
> Not sure this is a problem.
> {quote}
> 2012-04-26 23:15:51,763 WARN hdfs.DFSClient (DFSOutputStream.java:run(710))
> - DFSOutputStream ResponseProcessor exception for block
> BP-1372255573-49.249.124.17-1335462329685:blk_-843504080180201_1005
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:95)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:657)
> Exception in thread "DataXceiver for client /127.0.0.1:52323 [Cleaning up]"
> java.lang.NullPointerException
> at org.apache.hadoop.ipc.Server$Listener.getAddress(Server.java:669)
> at org.apache.hadoop.ipc.Server.getListenerAddress(Server.java:1988)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getIpcPort(DataNode.java:882)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getDisplayName(DataNode.java:863)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:171)
> at java.lang.Thread.run(Unknown Source){quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2802) Support for RW/RO snapshots in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271497#comment-13271497 ] Eli Collins commented on HDFS-2802: --- Hey Hari, Could you elaborate some on scenario #3? The motivation is test, so I assume use cases are things like (1) giving an experimental Hive/Pig user safe access /datasets/xyz or (2) running a test/dev HBase instance from an existing /hbase directory. Use case #1 seems like it could be accomplished in practice with external tables (or, in general, analysis programs that can separate their input data from directories they create/mutate). Use case #2 seems unlikely to be used in practice. Could be useful if /hbase is production (ie big) but in that case you wouldn't want to run a dev/test HBase build against a production HDFS cluster (as the production HBase instance runs there as well). Ie the usual case here is that you have a separate test/dev HBase/HDFS instance and you need/want to copy the datasets. What's your plan for handling append and truncate? The design document does not discuss whether you're punting on these, or if not the semantics (eg for a snapshot taken while a block is being written) or the implementation. There's relevant design and implementation discussion in HDFS-233 and [Snapshots in Hadoop Distributed File System|http://www.cs.berkeley.edu/~sameerag/hdfs-snapshots.pdf] that's worth referencing. Looking forward to checking out your prototype patch. Thanks, Eli > Support for RW/RO snapshots in HDFS > --- > > Key: HDFS-2802 > URL: https://issues.apache.org/jira/browse/HDFS-2802 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node, name-node >Affects Versions: 0.24.0 >Reporter: Hari Mankude >Assignee: Hari Mankude > Attachments: snapshot-one-pager.pdf > > > Snapshots are point in time images of parts of the filesystem or the entire > filesystem. Snapshots can be a read-only or a read-write point in time copy > of the filesystem. There are several use cases for snapshots in HDFS. I will > post a detailed write-up soon with with more information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3391) Failing tests in branch-2
[ https://issues.apache.org/jira/browse/HDFS-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271511#comment-13271511 ] John George commented on HDFS-3391: --- I was wrong about trunk - I had not updated to the latest trunk and once I did, I could reproduce this consistently. It is easily reproducible on branch-2.0.0-alpha as well. I have closed it down to the changes in HDFS-3157. Without HDFS-3157, the test passes consistently and with the patch, it fails. I am going to unassign myself from the JIRA and going to ping HDFS-3157 > Failing tests in branch-2 > - > > Key: HDFS-3391 > URL: https://issues.apache.org/jira/browse/HDFS-3391 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Arun C Murthy >Assignee: John George >Priority: Critical > > Running org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.208 sec <<< > FAILURE! > -- > Running org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover > Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 81.195 sec > <<< FAILURE! > -- -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3391) Failing tests in branch-2
[ https://issues.apache.org/jira/browse/HDFS-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated HDFS-3391: -- Assignee: (was: John George) > Failing tests in branch-2 > - > > Key: HDFS-3391 > URL: https://issues.apache.org/jira/browse/HDFS-3391 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Arun C Murthy >Priority: Critical > > Running org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.208 sec <<< > FAILURE! > -- > Running org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover > Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 81.195 sec > <<< FAILURE! > -- -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3157) Error in deleting block is keep on coming from DN even after the block report and directory scanning has happened
[
https://issues.apache.org/jira/browse/HDFS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271514#comment-13271514
]
John George commented on HDFS-3157:
---
I believe this JIRA broke TestPipelinesFailover as stated in HDFS-3391. Could
you take a look?
> Error in deleting block is keep on coming from DN even after the block report
> and directory scanning has happened
> -
>
> Key: HDFS-3157
> URL: https://issues.apache.org/jira/browse/HDFS-3157
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: J.Andreina
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HDFS-3157.patch, HDFS-3157.patch, HDFS-3157.patch
>
>
> Cluster setup:
> 1NN,Three DN(DN1,DN2,DN3),replication factor-2,"dfs.blockreport.intervalMsec"
> 300,"dfs.datanode.directoryscan.interval" 1
> step 1: write one file "a.txt" with sync(not closed)
> step 2: Delete the blocks in one of the datanode say DN1(from rbw) to which
> replication happened.
> step 3: close the file.
> Since the replication factor is 2 the blocks are replicated to the other
> datanode.
> Then at the NN side the following cmd is issued to DN from which the block is
> deleted
> -
> {noformat}
> 2012-03-19 13:41:36,905 INFO org.apache.hadoop.hdfs.StateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: duplicate requested for
> blk_2903555284838653156 to add as corrupt on XX.XX.XX.XX by /XX.XX.XX.XX
> because reported RBW replica with genstamp 1002 does not match COMPLETE
> block's genstamp in block map 1003
> 2012-03-19 13:41:39,588 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> Removing block blk_2903555284838653156_1003 from neededReplications as it has
> enough replicas.
> {noformat}
> From the datanode side in which the block is deleted the following exception
> occured
> {noformat}
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unexpected error trying to delete block blk_2903555284838653156_1003.
> BlockInfo not found in volumeMap.
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:2061)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:581)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:545)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:690)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:522)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:662)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3384) DataStreamer thread should be closed immediatly when failed to setup a PipelineForAppendOrRecovery
[
https://issues.apache.org/jira/browse/HDFS-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
amith updated HDFS-3384:
Attachment: HDFS-3384.patch
When a pipeline setup for append/recovery may fail. If the setup fails we are
setting streamerClosed but there is no condition which cause the streamer
thread to stop.
currently in the given stack trace
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:510)
{code}
caused has a side effect of not closing the streamer thread even after setting
the streamerClosed.
Manually verified the scenario
write a file
corrupt block manually
call append
> DataStreamer thread should be closed immediatly when failed to setup a
> PipelineForAppendOrRecovery
> --
>
> Key: HDFS-3384
> URL: https://issues.apache.org/jira/browse/HDFS-3384
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 2.0.0
>Reporter: Brahma Reddy Battula
>Assignee: amith
> Attachments: HDFS-3384.patch
>
>
> Scenraio:
> =
> write a file
> corrupt block manually
> call append..
> {noformat}
> 2012-04-19 09:33:10,776 INFO hdfs.DFSClient
> (DFSOutputStream.java:createBlockOutputStream(1059)) - Exception in
> createBlockOutputStream
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1039)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:939)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> 2012-04-19 09:33:10,807 WARN hdfs.DFSClient (DFSOutputStream.java:run(549))
> - DataStreamer Exception
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:510)
> 2012-04-19 09:33:10,807 WARN hdfs.DFSClient
> (DFSOutputStream.java:hflush(1511)) - Error while syncing
> java.io.IOException: All datanodes 10.18.40.20:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:908)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> java.io.IOException: All datanodes 10.18.40.20:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:908)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3384) DataStreamer thread should be closed immediatly when failed to setup a PipelineForAppendOrRecovery
[
https://issues.apache.org/jira/browse/HDFS-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
amith updated HDFS-3384:
Target Version/s: 3.0.0 (was: 2.0.0)
Status: Patch Available (was: Open)
> DataStreamer thread should be closed immediatly when failed to setup a
> PipelineForAppendOrRecovery
> --
>
> Key: HDFS-3384
> URL: https://issues.apache.org/jira/browse/HDFS-3384
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 2.0.0
>Reporter: Brahma Reddy Battula
>Assignee: amith
> Attachments: HDFS-3384.patch
>
>
> Scenraio:
> =
> write a file
> corrupt block manually
> call append..
> {noformat}
> 2012-04-19 09:33:10,776 INFO hdfs.DFSClient
> (DFSOutputStream.java:createBlockOutputStream(1059)) - Exception in
> createBlockOutputStream
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1039)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:939)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> 2012-04-19 09:33:10,807 WARN hdfs.DFSClient (DFSOutputStream.java:run(549))
> - DataStreamer Exception
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:510)
> 2012-04-19 09:33:10,807 WARN hdfs.DFSClient
> (DFSOutputStream.java:hflush(1511)) - Error while syncing
> java.io.IOException: All datanodes 10.18.40.20:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:908)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> java.io.IOException: All datanodes 10.18.40.20:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:908)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3369) change variable names referring to inode in blockmanagement to more appropriate
[ https://issues.apache.org/jira/browse/HDFS-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated HDFS-3369: -- Attachment: HDFS-3369.patch Both the hdfs changes and mapreduce (raid) changes are included in this patch. > change variable names referring to inode in blockmanagement to more > appropriate > --- > > Key: HDFS-3369 > URL: https://issues.apache.org/jira/browse/HDFS-3369 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 2.0.0, 3.0.0 >Reporter: John George >Assignee: John George >Priority: Minor > Attachments: HDFS-3369.patch > > > We should rename BlocksMap.getINode(..) and, in addition, the local variable > names such as fileInode to match 'block collection' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3369) change variable names referring to inode in blockmanagement to more appropriate
[ https://issues.apache.org/jira/browse/HDFS-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated HDFS-3369: -- Target Version/s: 2.0.0, 3.0.0 (was: 3.0.0, 2.0.0) Status: Patch Available (was: Open) > change variable names referring to inode in blockmanagement to more > appropriate > --- > > Key: HDFS-3369 > URL: https://issues.apache.org/jira/browse/HDFS-3369 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 2.0.0, 3.0.0 >Reporter: John George >Assignee: John George >Priority: Minor > Attachments: HDFS-3369.patch > > > We should rename BlocksMap.getINode(..) and, in addition, the local variable > names such as fileInode to match 'block collection' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2994) If lease is recovered successfully inline with create, create can fail
[
https://issues.apache.org/jira/browse/HDFS-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
amith reassigned HDFS-2994:
---
Assignee: amith
> If lease is recovered successfully inline with create, create can fail
> --
>
> Key: HDFS-2994
> URL: https://issues.apache.org/jira/browse/HDFS-2994
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.24.0
>Reporter: Todd Lipcon
>Assignee: amith
>
> I saw the following logs on my test cluster:
> {code}
> 2012-02-22 14:35:22,887 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: startFile: recover lease
> [Lease. Holder: DFSClient_attempt_1329943893604_0007_m_000376_0_453973131_1,
> pendingcreates: 1], src=/benchmarks/TestDFSIO/io_data/test_io_6 from client
> DFSClient_attempt_1329943893604_0007_m_000376_0_453973131_1
> 2012-02-22 14:35:22,887 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering lease=[Lease.
> Holder: DFSClient_attempt_1329943893604_0007_m_000376_0_453973131_1,
> pendingcreates: 1], src=/benchmarks/TestDFSIO/io_data/test_io_6
> 2012-02-22 14:35:22,888 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> internalReleaseLease: All existing blocks are COMPLETE, lease removed, file
> closed.
> 2012-02-22 14:35:22,888 WARN org.apache.hadoop.hdfs.StateChange: DIR*
> FSDirectory.replaceNode: failed to remove
> /benchmarks/TestDFSIO/io_data/test_io_6
> 2012-02-22 14:35:22,888 WARN org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.startFile: FSDirectory.replaceNode: failed to remove
> /benchmarks/TestDFSIO/io_data/test_io_6
> {code}
> It seems like, if {{recoverLeaseInternal}} succeeds in {{startFileInternal}},
> then the INode will be replaced with a new one, meaning the later
> {{replaceNode}} call can fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3369) change variable names referring to inode in blockmanagement to more appropriate
[ https://issues.apache.org/jira/browse/HDFS-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271535#comment-13271535 ] John George commented on HDFS-3369: --- I ran the hdfs test suite to ensure that nothing broke because of this change. I also ran the specific raid changes. > change variable names referring to inode in blockmanagement to more > appropriate > --- > > Key: HDFS-3369 > URL: https://issues.apache.org/jira/browse/HDFS-3369 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 2.0.0, 3.0.0 >Reporter: John George >Assignee: John George >Priority: Minor > Attachments: HDFS-3369.patch > > > We should rename BlocksMap.getINode(..) and, in addition, the local variable > names such as fileInode to match 'block collection' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3369) change variable names referring to inode in blockmanagement to more appropriate
[ https://issues.apache.org/jira/browse/HDFS-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271545#comment-13271545 ] Hadoop QA commented on HDFS-3369: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526180/HDFS-3369.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javadoc. The javadoc tool appears to have generated -2 warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2395//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2395//console This message is automatically generated. > change variable names referring to inode in blockmanagement to more > appropriate > --- > > Key: HDFS-3369 > URL: https://issues.apache.org/jira/browse/HDFS-3369 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: 2.0.0, 3.0.0 >Reporter: John George >Assignee: John George >Priority: Minor > Attachments: HDFS-3369.patch > > > We should rename BlocksMap.getINode(..) and, in addition, the local variable > names such as fileInode to match 'block collection' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3026) HA: Handle failure during HA state transition
[ https://issues.apache.org/jira/browse/HDFS-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3026: - Target Version/s: 2.0.0 (was: 0.24.0) Affects Version/s: (was: 0.24.0) 2.0.0 > HA: Handle failure during HA state transition > - > > Key: HDFS-3026 > URL: https://issues.apache.org/jira/browse/HDFS-3026 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3026-HDFS-1623.patch > > > This JIRA is to address a TODO in NameNode about handling the possibility of > an incomplete HA state transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2802) Support for RW/RO snapshots in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271616#comment-13271616 ] Hari Mankude commented on HDFS-2802: @Eli, Regarding scenario #3, consider a hbase setup with huge dataset in production. A new app has been developed which needs to be validated against production dataset. It is not feasible to copy the entire dataset to a test setup. At the same time, app is not ready for production and it is not safe to have the app modify the data in the production database. One of the solutions for these types of problems is to take a RW snapshot of the production dataset and then have the development app run against the RW snapshot. After the app testing is done, RW snap is deleted. This assumes that the cluster has sufficient compute capacity and incremental storage capacity to support RW snaps. Regarding appends, current prototype of snapshot relies on the filesize that is available at the namenode. So, if a file is appended after snap is taken, then it is a no-op from a snap perspective. If a snap is taken of a file which has append pipeline setup, inode is of type underconstruction in the NN. Prototype relies on filesize that is available on the NN for snaps. This might not be perfect and I have some ideas on trying to acquire more upto-date filesize. I thought that truncate is not supported currently in the trunk. If you are referring to deletes, prototype handles deletes correctly without issues. I will post a more detailed doc after I am done with HA related work. > Support for RW/RO snapshots in HDFS > --- > > Key: HDFS-2802 > URL: https://issues.apache.org/jira/browse/HDFS-2802 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node, name-node >Affects Versions: 0.24.0 >Reporter: Hari Mankude >Assignee: Hari Mankude > Attachments: snapshot-one-pager.pdf > > > Snapshots are point in time images of parts of the filesystem or the entire > filesystem. Snapshots can be a read-only or a read-write point in time copy > of the filesystem. There are several use cases for snapshots in HDFS. I will > post a detailed write-up soon with with more information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3384) DataStreamer thread should be closed immediatly when failed to setup a PipelineForAppendOrRecovery
[
https://issues.apache.org/jira/browse/HDFS-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271619#comment-13271619
]
Hadoop QA commented on HDFS-3384:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12526178/HDFS-3384.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/2396//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2396//console
This message is automatically generated.
> DataStreamer thread should be closed immediatly when failed to setup a
> PipelineForAppendOrRecovery
> --
>
> Key: HDFS-3384
> URL: https://issues.apache.org/jira/browse/HDFS-3384
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 2.0.0
>Reporter: Brahma Reddy Battula
>Assignee: amith
> Attachments: HDFS-3384.patch
>
>
> Scenraio:
> =
> write a file
> corrupt block manually
> call append..
> {noformat}
> 2012-04-19 09:33:10,776 INFO hdfs.DFSClient
> (DFSOutputStream.java:createBlockOutputStream(1059)) - Exception in
> createBlockOutputStream
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1039)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:939)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> 2012-04-19 09:33:10,807 WARN hdfs.DFSClient (DFSOutputStream.java:run(549))
> - DataStreamer Exception
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:510)
> 2012-04-19 09:33:10,807 WARN hdfs.DFSClient
> (DFSOutputStream.java:hflush(1511)) - Error while syncing
> java.io.IOException: All datanodes 10.18.40.20:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:908)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> java.io.IOException: All datanodes 10.18.40.20:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:908)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3390) DFSAdmin should print full stack traces of errors when DEBUG logging is enabled
[ https://issues.apache.org/jira/browse/HDFS-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3390: - Hadoop Flags: Reviewed +1 patch looks good. > DFSAdmin should print full stack traces of errors when DEBUG logging is > enabled > --- > > Key: HDFS-3390 > URL: https://issues.apache.org/jira/browse/HDFS-3390 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Minor > Attachments: HDFS-3390.patch, HDFS-3390.patch > > > If an error is encountered when running an `hdfs dfsadmin ...' command, only > the exception's message is output. It would be handy for debugging if the > full stack trace of the exception were output when DEBUG logging is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3157) Error in deleting block is keep on coming from DN even after the block report and directory scanning has happened
[
https://issues.apache.org/jira/browse/HDFS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271639#comment-13271639
]
Uma Maheswara Rao G commented on HDFS-3157:
---
Hi John, Thanks for digging into HDFS-3391. I am wondering, we got +1 here from
QA.
Anyway, I have discussed with Ashish, for taking a look. Let him find the
actual cause for it.
> Error in deleting block is keep on coming from DN even after the block report
> and directory scanning has happened
> -
>
> Key: HDFS-3157
> URL: https://issues.apache.org/jira/browse/HDFS-3157
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: J.Andreina
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HDFS-3157.patch, HDFS-3157.patch, HDFS-3157.patch
>
>
> Cluster setup:
> 1NN,Three DN(DN1,DN2,DN3),replication factor-2,"dfs.blockreport.intervalMsec"
> 300,"dfs.datanode.directoryscan.interval" 1
> step 1: write one file "a.txt" with sync(not closed)
> step 2: Delete the blocks in one of the datanode say DN1(from rbw) to which
> replication happened.
> step 3: close the file.
> Since the replication factor is 2 the blocks are replicated to the other
> datanode.
> Then at the NN side the following cmd is issued to DN from which the block is
> deleted
> -
> {noformat}
> 2012-03-19 13:41:36,905 INFO org.apache.hadoop.hdfs.StateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: duplicate requested for
> blk_2903555284838653156 to add as corrupt on XX.XX.XX.XX by /XX.XX.XX.XX
> because reported RBW replica with genstamp 1002 does not match COMPLETE
> block's genstamp in block map 1003
> 2012-03-19 13:41:39,588 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> Removing block blk_2903555284838653156_1003 from neededReplications as it has
> enough replicas.
> {noformat}
> From the datanode side in which the block is deleted the following exception
> occured
> {noformat}
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unexpected error trying to delete block blk_2903555284838653156_1003.
> BlockInfo not found in volumeMap.
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:2061)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:581)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:545)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:690)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:522)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:662)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3370) HDFS hardlink
[ https://issues.apache.org/jira/browse/HDFS-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liyin Tang updated HDFS-3370: - Attachment: (was: HDFS-HardLink.pdf) > HDFS hardlink > - > > Key: HDFS-3370 > URL: https://issues.apache.org/jira/browse/HDFS-3370 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Hairong Kuang >Assignee: Liyin Tang > > We'd like to add a new feature hardlink to HDFS that allows harlinked files > to share data without copying. Currently we will support hardlinking only > closed files, but it could be extended to unclosed files as well. > Among many potential use cases of the feature, the following two are > primarily used in facebook: > 1. This provides a lightweight way for applications like hbase to create a > snapshot; > 2. This also allows an application like Hive to move a table to a different > directory without breaking current running hive queries. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3370) HDFS hardlink
[ https://issues.apache.org/jira/browse/HDFS-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liyin Tang updated HDFS-3370: - Attachment: HDFS-HardLink.pdf > HDFS hardlink > - > > Key: HDFS-3370 > URL: https://issues.apache.org/jira/browse/HDFS-3370 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Hairong Kuang >Assignee: Liyin Tang > Attachments: HDFS-HardLink.pdf > > > We'd like to add a new feature hardlink to HDFS that allows harlinked files > to share data without copying. Currently we will support hardlinking only > closed files, but it could be extended to unclosed files as well. > Among many potential use cases of the feature, the following two are > primarily used in facebook: > 1. This provides a lightweight way for applications like hbase to create a > snapshot; > 2. This also allows an application like Hive to move a table to a different > directory without breaking current running hive queries. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3370) HDFS hardlink
[ https://issues.apache.org/jira/browse/HDFS-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271684#comment-13271684 ] Liyin Tang commented on HDFS-3370: -- @M.C.Srivas, I am afraid that I didn't quite understand your concerns. bq. 1. Changing the permissions along the path "/path1/dirA" to make "file" inaccessible works, but now with hard-links "/path2/dirB" is wide open. 2. Rename "/path2/dirB" to "/path3/dirC" will require taking locks on "/path1/dirA" ... but the "file" does not have "parent ptrs" to figure out which path(s) to lock. 1) Hardlinked files are suposed to have the same permission as the source file. 2) Each INodeFile do have a parent pointer to its parent in HDFS and also which lock are you talking about exactly (in the implementation's perspective) ? @Daryn Sharp: bq. I did a little testing and didn't realize that a hard link retains the same attrs (owner, group, perms, xattrs, etc) as the original file. Changing one implicit changes the others, so that negates some issues such as differing replication factor concerns. Perhaps hard link creation can be restricted to only the file owner and superuser. Totally agreed:) bq. The quota concerns are still a bit more complex. Unixy systems like linux and bsd only have fs level quotas for users, so quota handling is trivial compared to directory level quotas in hdfs. Since all hard links implicitly have the same owner, quotas are as simple as incrementing the user's ds quota is at file creation, and decrement when all links are removed. This why a DOS is possible against a user. >From the security perspective, users should be responsible to set the correct >permission to protect themselves. In this cases, users should ONLY grant the >EXECUTE permission to the trusted users for hardlinkling. bq. I'm sorry if I'm missing a detail, but I remain unclear on how you are proposing to handle the directory level quotas. I don't fully grok how finding a common ancestor with a quota is sufficient because quotas can be added or removed at any time. Maybe part of the issue too is I have nested directories with individual quotas in mind, whereas maybe you are assuming one and only one quota from the root? Would you mind giving me an exact example for your concerns about quotas? I would be very happy to explain it in details :) > HDFS hardlink > - > > Key: HDFS-3370 > URL: https://issues.apache.org/jira/browse/HDFS-3370 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Hairong Kuang >Assignee: Liyin Tang > Attachments: HDFS-HardLink.pdf > > > We'd like to add a new feature hardlink to HDFS that allows harlinked files > to share data without copying. Currently we will support hardlinking only > closed files, but it could be extended to unclosed files as well. > Among many potential use cases of the feature, the following two are > primarily used in facebook: > 1. This provides a lightweight way for applications like hbase to create a > snapshot; > 2. This also allows an application like Hive to move a table to a different > directory without breaking current running hive queries. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3390) DFSAdmin should print full stack traces of errors when DEBUG logging is enabled
[ https://issues.apache.org/jira/browse/HDFS-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271686#comment-13271686 ] Hudson commented on HDFS-3390: -- Integrated in Hadoop-Common-trunk-Commit #2213 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2213/]) HDFS-3390. DFSAdmin should print full stack traces of errors when DEBUG logging is enabled. Contributed by Aaron T. Myers. (Revision 1336324) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336324 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java > DFSAdmin should print full stack traces of errors when DEBUG logging is > enabled > --- > > Key: HDFS-3390 > URL: https://issues.apache.org/jira/browse/HDFS-3390 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Minor > Attachments: HDFS-3390.patch, HDFS-3390.patch > > > If an error is encountered when running an `hdfs dfsadmin ...' command, only > the exception's message is output. It would be handy for debugging if the > full stack trace of the exception were output when DEBUG logging is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3390) DFSAdmin should print full stack traces of errors when DEBUG logging is enabled
[ https://issues.apache.org/jira/browse/HDFS-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3390: - Resolution: Fixed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) Thanks a lot for the reviews, Nicholas. I've just committed this to trunk and branch-2. > DFSAdmin should print full stack traces of errors when DEBUG logging is > enabled > --- > > Key: HDFS-3390 > URL: https://issues.apache.org/jira/browse/HDFS-3390 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Minor > Fix For: 2.0.0 > > Attachments: HDFS-3390.patch, HDFS-3390.patch > > > If an error is encountered when running an `hdfs dfsadmin ...' command, only > the exception's message is output. It would be handy for debugging if the > full stack trace of the exception were output when DEBUG logging is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3390) DFSAdmin should print full stack traces of errors when DEBUG logging is enabled
[ https://issues.apache.org/jira/browse/HDFS-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271692#comment-13271692 ] Hudson commented on HDFS-3390: -- Integrated in Hadoop-Hdfs-trunk-Commit #2288 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2288/]) HDFS-3390. DFSAdmin should print full stack traces of errors when DEBUG logging is enabled. Contributed by Aaron T. Myers. (Revision 1336324) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336324 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java > DFSAdmin should print full stack traces of errors when DEBUG logging is > enabled > --- > > Key: HDFS-3390 > URL: https://issues.apache.org/jira/browse/HDFS-3390 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Minor > Fix For: 2.0.0 > > Attachments: HDFS-3390.patch, HDFS-3390.patch > > > If an error is encountered when running an `hdfs dfsadmin ...' command, only > the exception's message is output. It would be handy for debugging if the > full stack trace of the exception were output when DEBUG logging is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3390) DFSAdmin should print full stack traces of errors when DEBUG logging is enabled
[ https://issues.apache.org/jira/browse/HDFS-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271713#comment-13271713 ] Hudson commented on HDFS-3390: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2230 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2230/]) HDFS-3390. DFSAdmin should print full stack traces of errors when DEBUG logging is enabled. Contributed by Aaron T. Myers. (Revision 1336324) Result = ABORTED atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336324 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java > DFSAdmin should print full stack traces of errors when DEBUG logging is > enabled > --- > > Key: HDFS-3390 > URL: https://issues.apache.org/jira/browse/HDFS-3390 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Minor > Fix For: 2.0.0 > > Attachments: HDFS-3390.patch, HDFS-3390.patch > > > If an error is encountered when running an `hdfs dfsadmin ...' command, only > the exception's message is output. It would be handy for debugging if the > full stack trace of the exception were output when DEBUG logging is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3157) Error in deleting block is keep on coming from DN even after the block report and directory scanning has happened
[
https://issues.apache.org/jira/browse/HDFS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271725#comment-13271725
]
Tsz Wo (Nicholas), SZE commented on HDFS-3157:
--
When replication happens, somehow the replica already exists.
{noformat}
//TestRBWBlockInvalidation output
2012-05-09 12:30:04,122 INFO datanode.DataNode
(DataXceiver.java:writeBlock(495))
- opWriteBlock
BP-2087796974-10.10.11.90-1336591801017:blk_-571802999240948417_1003 received
exception org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException:
Block BP-2087796974-10.10.11.90-1336591801017:blk_-571802999240948417_1003
already exists in state RBW and thus cannot be created.
{noformat}
> Error in deleting block is keep on coming from DN even after the block report
> and directory scanning has happened
> -
>
> Key: HDFS-3157
> URL: https://issues.apache.org/jira/browse/HDFS-3157
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: J.Andreina
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HDFS-3157.patch, HDFS-3157.patch, HDFS-3157.patch
>
>
> Cluster setup:
> 1NN,Three DN(DN1,DN2,DN3),replication factor-2,"dfs.blockreport.intervalMsec"
> 300,"dfs.datanode.directoryscan.interval" 1
> step 1: write one file "a.txt" with sync(not closed)
> step 2: Delete the blocks in one of the datanode say DN1(from rbw) to which
> replication happened.
> step 3: close the file.
> Since the replication factor is 2 the blocks are replicated to the other
> datanode.
> Then at the NN side the following cmd is issued to DN from which the block is
> deleted
> -
> {noformat}
> 2012-03-19 13:41:36,905 INFO org.apache.hadoop.hdfs.StateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: duplicate requested for
> blk_2903555284838653156 to add as corrupt on XX.XX.XX.XX by /XX.XX.XX.XX
> because reported RBW replica with genstamp 1002 does not match COMPLETE
> block's genstamp in block map 1003
> 2012-03-19 13:41:39,588 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> Removing block blk_2903555284838653156_1003 from neededReplications as it has
> enough replicas.
> {noformat}
> From the datanode side in which the block is deleted the following exception
> occured
> {noformat}
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unexpected error trying to delete block blk_2903555284838653156_1003.
> BlockInfo not found in volumeMap.
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:2061)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:581)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:545)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:690)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:522)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:662)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3157) Error in deleting block is keep on coming from DN even after the block report and directory scanning has happened
[
https://issues.apache.org/jira/browse/HDFS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271736#comment-13271736
]
Tsz Wo (Nicholas), SZE commented on HDFS-3157:
--
Here is the reason of the TestRBWBlockInvalidation failing:
The block and meta files are deleted in dn0 but it is still in the replica map
(FsDatasetImpl.volumeMap). When replication happens, it fails since the block
is still in the replica map and so it throw ReplicaAlreadyExistsException.
Therefore, the number of live replicas remains 2.
> ... I am wondering, we got +1 here from QA.
I also don't understand that why Jenkins has +1'ed on it. It seems that the
test must always fail.
> Error in deleting block is keep on coming from DN even after the block report
> and directory scanning has happened
> -
>
> Key: HDFS-3157
> URL: https://issues.apache.org/jira/browse/HDFS-3157
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: J.Andreina
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HDFS-3157.patch, HDFS-3157.patch, HDFS-3157.patch
>
>
> Cluster setup:
> 1NN,Three DN(DN1,DN2,DN3),replication factor-2,"dfs.blockreport.intervalMsec"
> 300,"dfs.datanode.directoryscan.interval" 1
> step 1: write one file "a.txt" with sync(not closed)
> step 2: Delete the blocks in one of the datanode say DN1(from rbw) to which
> replication happened.
> step 3: close the file.
> Since the replication factor is 2 the blocks are replicated to the other
> datanode.
> Then at the NN side the following cmd is issued to DN from which the block is
> deleted
> -
> {noformat}
> 2012-03-19 13:41:36,905 INFO org.apache.hadoop.hdfs.StateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: duplicate requested for
> blk_2903555284838653156 to add as corrupt on XX.XX.XX.XX by /XX.XX.XX.XX
> because reported RBW replica with genstamp 1002 does not match COMPLETE
> block's genstamp in block map 1003
> 2012-03-19 13:41:39,588 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> Removing block blk_2903555284838653156_1003 from neededReplications as it has
> enough replicas.
> {noformat}
> From the datanode side in which the block is deleted the following exception
> occured
> {noformat}
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unexpected error trying to delete block blk_2903555284838653156_1003.
> BlockInfo not found in volumeMap.
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:2061)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:581)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:545)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:690)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:522)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:662)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3092) Enable journal protocol based editlog streaming for standby namenode
[ https://issues.apache.org/jira/browse/HDFS-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271737#comment-13271737 ] Nathan Roberts commented on HDFS-3092: -- Eli, thanks for the writeup. One question on this statement "(ie option 1 is fundamentally slower)." We already double-buffer the editstreams in the namenode so isn't it true that users will only see latency effects if the buffering isn't able to keep up? In other words, isn't it the case that as long as the slowest journal is keeping up with demand, there's no significant difference in performance? > Enable journal protocol based editlog streaming for standby namenode > > > Key: HDFS-3092 > URL: https://issues.apache.org/jira/browse/HDFS-3092 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, name-node >Affects Versions: 0.24.0, 0.23.3 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: ComparisonofApproachesforHAJournals.pdf, JNStates.png, > MultipleSharedJournals.pdf, MultipleSharedJournals.pdf, > MultipleSharedJournals.pdf, Removingfilerdependency.pdf > > > Currently standby namenode relies on reading shared editlogs to stay current > with the active namenode, for namespace changes. BackupNode used streaming > edits from active namenode for doing the same. This jira is to explore using > journal protocol based editlog streams for the standby namenode. A daemon in > standby will get the editlogs from the active and write it to local edits. To > begin with, the existing standby mechanism of reading from a file, will > continue to be used, instead of from shared edits, from the local edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3385) ClassCastException when trying to append a file
[
https://issues.apache.org/jira/browse/HDFS-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271753#comment-13271753
]
Tsz Wo (Nicholas), SZE commented on HDFS-3385:
--
Thanks, Brahma. The last block of INodeFileUnderConstruction is not
necessarily a BlockInfoUnderConstruction. I will try to add a test.
We should also remove the generic from INodeFile.getLastBlock(). It is useless
as shown here.
> ClassCastException when trying to append a file
> ---
>
> Key: HDFS-3385
> URL: https://issues.apache.org/jira/browse/HDFS-3385
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 3.0.0
> Environment: HDFS
>Reporter: Brahma Reddy Battula
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: h3385_20120508.patch
>
>
> When I try to append a file I got
> {noformat}
> 2012-05-08 18:13:40,506 WARN util.KerberosName
> (KerberosName.java:(87)) - Kerberos krb5 configuration not found,
> setting default realm to empty
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo cannot be cast to
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoUnderConstruction
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1787)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1584)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1824)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:425)
> ...
> at org.apache.hadoop.hdfs.DFSClient.callAppend(DFSClient.java:1150)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1189)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1177)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:221)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:1)
> at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:981)
> at
> org.apache.hadoop.hdfs.server.datanode.DeleteMe.main(DeleteMe.java:26)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3385) ClassCastException when trying to append a file
[
https://issues.apache.org/jira/browse/HDFS-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-3385:
-
Priority: Major (was: Minor)
Affects Version/s: (was: 3.0.0)
Fix Version/s: (was: 3.0.0)
Assignee: Tsz Wo (Nicholas), SZE
> ClassCastException when trying to append a file
> ---
>
> Key: HDFS-3385
> URL: https://issues.apache.org/jira/browse/HDFS-3385
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Environment: HDFS
>Reporter: Brahma Reddy Battula
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h3385_20120508.patch
>
>
> When I try to append a file I got
> {noformat}
> 2012-05-08 18:13:40,506 WARN util.KerberosName
> (KerberosName.java:(87)) - Kerberos krb5 configuration not found,
> setting default realm to empty
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo cannot be cast to
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoUnderConstruction
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1787)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1584)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1824)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:425)
> ...
> at org.apache.hadoop.hdfs.DFSClient.callAppend(DFSClient.java:1150)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1189)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1177)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:221)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:1)
> at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:981)
> at
> org.apache.hadoop.hdfs.server.datanode.DeleteMe.main(DeleteMe.java:26)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3157) Error in deleting block is keep on coming from DN even after the block report and directory scanning has happened
[
https://issues.apache.org/jira/browse/HDFS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271758#comment-13271758
]
Tsz Wo (Nicholas), SZE commented on HDFS-3157:
--
I think we only have to fix the test. The behavior of dn0 is expected.
DirectoryScanner.reconcile() should be able to remove the block from the
replica map later on.
> Error in deleting block is keep on coming from DN even after the block report
> and directory scanning has happened
> -
>
> Key: HDFS-3157
> URL: https://issues.apache.org/jira/browse/HDFS-3157
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: J.Andreina
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HDFS-3157.patch, HDFS-3157.patch, HDFS-3157.patch
>
>
> Cluster setup:
> 1NN,Three DN(DN1,DN2,DN3),replication factor-2,"dfs.blockreport.intervalMsec"
> 300,"dfs.datanode.directoryscan.interval" 1
> step 1: write one file "a.txt" with sync(not closed)
> step 2: Delete the blocks in one of the datanode say DN1(from rbw) to which
> replication happened.
> step 3: close the file.
> Since the replication factor is 2 the blocks are replicated to the other
> datanode.
> Then at the NN side the following cmd is issued to DN from which the block is
> deleted
> -
> {noformat}
> 2012-03-19 13:41:36,905 INFO org.apache.hadoop.hdfs.StateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: duplicate requested for
> blk_2903555284838653156 to add as corrupt on XX.XX.XX.XX by /XX.XX.XX.XX
> because reported RBW replica with genstamp 1002 does not match COMPLETE
> block's genstamp in block map 1003
> 2012-03-19 13:41:39,588 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> Removing block blk_2903555284838653156_1003 from neededReplications as it has
> enough replicas.
> {noformat}
> From the datanode side in which the block is deleted the following exception
> occured
> {noformat}
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unexpected error trying to delete block blk_2903555284838653156_1003.
> BlockInfo not found in volumeMap.
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:2061)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:581)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:545)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:690)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:522)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:662)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3134) harden edit log loader against malformed or malicious input
[
https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-3134:
---
Attachment: HDFS-3134.009.patch
* fix a few more instances of the old
{code}
int length = in.readInt()
Ty myArray[] = new Ty[length];
{code}
> harden edit log loader against malformed or malicious input
> ---
>
> Key: HDFS-3134
> URL: https://issues.apache.org/jira/browse/HDFS-3134
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch,
> HDFS-3134.003.patch, HDFS-3134.004.patch, HDFS-3134.005.patch,
> HDFS-3134.006.patch, HDFS-3134.007.patch, HDFS-3134.009.patch
>
>
> Currently, the edit log loader does not handle bad or malicious input
> sensibly.
> We can often cause OutOfMemory exceptions, null pointer exceptions, or other
> unchecked exceptions to be thrown by feeding the edit log loader bad input.
> In some environments, an out of memory error can cause the JVM process to be
> terminated.
> It's clear that we want these exceptions to be thrown as IOException instead
> of as unchecked exceptions. We also want to avoid out of memory situations.
> The main task here is to put a sensible upper limit on the lengths of arrays
> and strings we allocate on command. The other task is to try to avoid
> creating unchecked exceptions (by dereferencing potentially-NULL pointers,
> for example). Instead, we should verify ahead of time and give a more
> sensible error message that reflects the problem with the input.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2631) Rewrite fuse-dfs to use the webhdfs protocol
[ https://issues.apache.org/jira/browse/HDFS-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaimin D Jetly updated HDFS-2631: - Attachment: HDFS-2631.1.patch This is an initial patch of fuse-webhdfs contrib project. Work is under progress. you can review the code and read the README file for the installation . Project supports all directory level commands and limited file level commands like read (more), write (indirection >), append (>>), find. Other functions that could be implemented in the future is statfs , symlink , access. For now one needs to manually download libcurl and libjson libraries and configure+make (refer README). But I will get it done through ant in the next patch. > Rewrite fuse-dfs to use the webhdfs protocol > > > Key: HDFS-2631 > URL: https://issues.apache.org/jira/browse/HDFS-2631 > Project: Hadoop HDFS > Issue Type: Improvement > Components: contrib/fuse-dfs >Reporter: Eli Collins >Assignee: Jaimin D Jetly > Attachments: HDFS-2631.1.patch > > > We should port the implementation of fuse-dfs to use the webhdfs protocol. > This has a number of benefits: > * Compatibility - allows a single fuse client to work across server versions > * Works with both WebHDFS and Hoop since they are protocol compatible > * Removes the overhead related to libhdfs (forking a jvm) > * Makes it easier to support features like security -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-3157) Error in deleting block is keep on coming from DN even after the block report and directory scanning has happened
[
https://issues.apache.org/jira/browse/HDFS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE reopened HDFS-3157:
--
The patch actually does not work since the INode in the new BlockInfo is not
set. Let's revert it and re-do the patch.
{noformat}
2012-05-09 16:05:29,960 INFO hdfs.StateChange
(BlockManager.java:markBlockAsCorrupt(926)) - BLOCK markBlockAsCorrupt:
block blk_-6891652617965059210_1002 could not be marked as corrupt as it does
not belong to any file
{noformat}
> Error in deleting block is keep on coming from DN even after the block report
> and directory scanning has happened
> -
>
> Key: HDFS-3157
> URL: https://issues.apache.org/jira/browse/HDFS-3157
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: J.Andreina
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HDFS-3157.patch, HDFS-3157.patch, HDFS-3157.patch
>
>
> Cluster setup:
> 1NN,Three DN(DN1,DN2,DN3),replication factor-2,"dfs.blockreport.intervalMsec"
> 300,"dfs.datanode.directoryscan.interval" 1
> step 1: write one file "a.txt" with sync(not closed)
> step 2: Delete the blocks in one of the datanode say DN1(from rbw) to which
> replication happened.
> step 3: close the file.
> Since the replication factor is 2 the blocks are replicated to the other
> datanode.
> Then at the NN side the following cmd is issued to DN from which the block is
> deleted
> -
> {noformat}
> 2012-03-19 13:41:36,905 INFO org.apache.hadoop.hdfs.StateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: duplicate requested for
> blk_2903555284838653156 to add as corrupt on XX.XX.XX.XX by /XX.XX.XX.XX
> because reported RBW replica with genstamp 1002 does not match COMPLETE
> block's genstamp in block map 1003
> 2012-03-19 13:41:39,588 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> Removing block blk_2903555284838653156_1003 from neededReplications as it has
> enough replicas.
> {noformat}
> From the datanode side in which the block is deleted the following exception
> occured
> {noformat}
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unexpected error trying to delete block blk_2903555284838653156_1003.
> BlockInfo not found in volumeMap.
> 2012-02-29 13:54:13,126 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Error processing datanode Command
> java.io.IOException: Error in deleting blocks.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.invalidate(FSDataset.java:2061)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:581)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:545)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:690)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:522)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:662)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3393) Rollback on newer release must not be allowed if the layout version in the snapshot is for older release.
Suresh Srinivas created HDFS-3393: - Summary: Rollback on newer release must not be allowed if the layout version in the snapshot is for older release. Key: HDFS-3393 URL: https://issues.apache.org/jira/browse/HDFS-3393 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 1.0.0, 2.0.0, 3.0.0 Reporter: Suresh Srinivas After upgrading to a new release, the procedure to rollback to older release is to start the namenode in older release with -rollback option. As reported in HADOOP-8371, the rollback was allowed on the newer release. During rollback, namenode must allow rollback only if the layout version of the namenode matches that of the layout version in the snapshot that the namenode is rolling back to. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3394) Do not use generic in INodeFile.getLastBlock()
Tsz Wo (Nicholas), SZE created HDFS-3394: Summary: Do not use generic in INodeFile.getLastBlock() Key: HDFS-3394 URL: https://issues.apache.org/jira/browse/HDFS-3394 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor As shown in HDFS-3385, the ClassCastException check in INodeFile.getLastBlock() is useless since generic type information is only available in compile-time but not run-time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271971#comment-13271971 ] Hadoop QA commented on HDFS-3134: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526231/HDFS-3134.009.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2397//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2397//console This message is automatically generated. > harden edit log loader against malformed or malicious input > --- > > Key: HDFS-3134 > URL: https://issues.apache.org/jira/browse/HDFS-3134 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.23.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, > HDFS-3134.003.patch, HDFS-3134.004.patch, HDFS-3134.005.patch, > HDFS-3134.006.patch, HDFS-3134.007.patch, HDFS-3134.009.patch > > > Currently, the edit log loader does not handle bad or malicious input > sensibly. > We can often cause OutOfMemory exceptions, null pointer exceptions, or other > unchecked exceptions to be thrown by feeding the edit log loader bad input. > In some environments, an out of memory error can cause the JVM process to be > terminated. > It's clear that we want these exceptions to be thrown as IOException instead > of as unchecked exceptions. We also want to avoid out of memory situations. > The main task here is to put a sensible upper limit on the lengths of arrays > and strings we allocate on command. The other task is to try to avoid > creating unchecked exceptions (by dereferencing potentially-NULL pointers, > for example). Instead, we should verify ahead of time and give a more > sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3391) Failing tests in branch-2
[
https://issues.apache.org/jira/browse/HDFS-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271979#comment-13271979
]
Tsz Wo (Nicholas), SZE commented on HDFS-3391:
--
For TestPipelinesFailover, there are also many "THIS IS NOT SUPPOSED TO HAPPEN"
as in [build
#2397|https://builds.apache.org/job/PreCommit-HDFS-Build/2397//testReport/org.apache.hadoop.hdfs.server.namenode.ha/TestPipelinesFailover/testLeaseRecoveryAfterFailover/].
{noformat}
2012-05-09 23:50:32,074 INFO ipc.Server (Server.java:run(1711)) - IPC Server
handler 2 on 46044,
call
org.apache.hadoop.hdfs.server.protocol.InterDatanodeProtocol.initReplicaRecovery
from 127.0.0.1:33818:
error: java.io.IOException: THIS IS NOT SUPPOSED TO HAPPEN:
replica.getGenerationStamp() >= recoveryId = 1006,
block=blk_-3039116449792967513_1003, replica=FinalizedReplica,
blk_-3039116449792967513_1006, FINALIZED
getNumBytes() = 2048
getBytesOnDisk() = 2048
getVisibleLength()= 2048
getVolume() =
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2/current
getBlockFile()=
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data2/current/BP-1244283508-67.195.138.20-1336607428359/current/finalized/blk_-3039116449792967513
unlinked =false
{noformat}
> Failing tests in branch-2
> -
>
> Key: HDFS-3391
> URL: https://issues.apache.org/jira/browse/HDFS-3391
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Arun C Murthy
>Priority: Critical
>
> Running org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.208 sec <<<
> FAILURE!
> --
> Running org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
> Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 81.195 sec
> <<< FAILURE!
> --
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3393) Rollback on newer release must not be allowed if the layout version in the snapshot is for older release.
[ https://issues.apache.org/jira/browse/HDFS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271988#comment-13271988 ] Todd Lipcon commented on HDFS-3393: --- Is this a dup of HDFS-1969? > Rollback on newer release must not be allowed if the layout version in the > snapshot is for older release. > - > > Key: HDFS-3393 > URL: https://issues.apache.org/jira/browse/HDFS-3393 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 1.0.0, 2.0.0, 3.0.0 >Reporter: Suresh Srinivas > > After upgrading to a new release, the procedure to rollback to older release > is to start the namenode in older release with -rollback option. > As reported in HADOOP-8371, the rollback was allowed on the newer release. > During rollback, namenode must allow rollback only if the layout version of > the namenode matches that of the layout version in the snapshot that the > namenode is rolling back to. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3341) Change minimum RPC versions to 2.0.0-SNAPSHOT instead of 2.0.0
[ https://issues.apache.org/jira/browse/HDFS-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-3341. --- Resolution: Fixed Fix Version/s: 2.0.0 Target Version/s: 2.0.0, 3.0.0 (was: 3.0.0, 2.0.0) Hadoop Flags: Reviewed > Change minimum RPC versions to 2.0.0-SNAPSHOT instead of 2.0.0 > -- > > Key: HDFS-3341 > URL: https://issues.apache.org/jira/browse/HDFS-3341 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0, 3.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 2.0.0 > > Attachments: hdfs-3341.txt > > > After we commit HADOOP-8340, tests will fail because the minimum version is > configured to be 2.0.0/3.0.0 instead of the corresponding SNAPSHOT builds. > When we commit HADOOP-8340, we should update the minimums at the same time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3341) Change minimum RPC versions to 2.0.0-SNAPSHOT instead of 2.0.0
[ https://issues.apache.org/jira/browse/HDFS-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271997#comment-13271997 ] Hudson commented on HDFS-3341: -- Integrated in Hadoop-Hdfs-trunk-Commit #2291 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2291/]) HDFS-3341, HADOOP-8340. SNAPSHOT build versions should compare as less than their eventual release. Contributed by Todd Lipcon. (Revision 1336459) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336459 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionUtil.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestVersionUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java > Change minimum RPC versions to 2.0.0-SNAPSHOT instead of 2.0.0 > -- > > Key: HDFS-3341 > URL: https://issues.apache.org/jira/browse/HDFS-3341 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0, 3.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 2.0.0 > > Attachments: hdfs-3341.txt > > > After we commit HADOOP-8340, tests will fail because the minimum version is > configured to be 2.0.0/3.0.0 instead of the corresponding SNAPSHOT builds. > When we commit HADOOP-8340, we should update the minimums at the same time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3341) Change minimum RPC versions to 2.0.0-SNAPSHOT instead of 2.0.0
[ https://issues.apache.org/jira/browse/HDFS-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271999#comment-13271999 ] Hudson commented on HDFS-3341: -- Integrated in Hadoop-Common-trunk-Commit #2216 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2216/]) HDFS-3341, HADOOP-8340. SNAPSHOT build versions should compare as less than their eventual release. Contributed by Todd Lipcon. (Revision 1336459) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336459 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionUtil.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestVersionUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java > Change minimum RPC versions to 2.0.0-SNAPSHOT instead of 2.0.0 > -- > > Key: HDFS-3341 > URL: https://issues.apache.org/jira/browse/HDFS-3341 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0, 3.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 2.0.0 > > Attachments: hdfs-3341.txt > > > After we commit HADOOP-8340, tests will fail because the minimum version is > configured to be 2.0.0/3.0.0 instead of the corresponding SNAPSHOT builds. > When we commit HADOOP-8340, we should update the minimums at the same time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1584) Need to check TGT and renew if needed when fetching delegation tokens using HFTP
[ https://issues.apache.org/jira/browse/HDFS-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-1584: --- Attachment: hdfs-1584-022.patch Patch for 22 is attached > Need to check TGT and renew if needed when fetching delegation tokens using > HFTP > > > Key: HDFS-1584 > URL: https://issues.apache.org/jira/browse/HDFS-1584 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h1584-01.patch, hdfs-1584-022.patch > > > Currently, there is no checking on TGT validity when calling > getDelegationToken(). The TGT may expire and the call will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1584) Need to check TGT and renew if needed when fetching delegation tokens using HFTP
[ https://issues.apache.org/jira/browse/HDFS-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272006#comment-13272006 ] Hadoop QA commented on HDFS-1584: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526262/hdfs-1584-022.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2398//console This message is automatically generated. > Need to check TGT and renew if needed when fetching delegation tokens using > HFTP > > > Key: HDFS-1584 > URL: https://issues.apache.org/jira/browse/HDFS-1584 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h1584-01.patch, hdfs-1584-022.patch > > > Currently, there is no checking on TGT validity when calling > getDelegationToken(). The TGT may expire and the call will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3341) Change minimum RPC versions to 2.0.0-SNAPSHOT instead of 2.0.0
[ https://issues.apache.org/jira/browse/HDFS-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272021#comment-13272021 ] Hudson commented on HDFS-3341: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2233 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2233/]) HDFS-3341, HADOOP-8340. SNAPSHOT build versions should compare as less than their eventual release. Contributed by Todd Lipcon. (Revision 1336459) Result = ABORTED todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336459 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionUtil.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestVersionUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java > Change minimum RPC versions to 2.0.0-SNAPSHOT instead of 2.0.0 > -- > > Key: HDFS-3341 > URL: https://issues.apache.org/jira/browse/HDFS-3341 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0, 3.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 2.0.0 > > Attachments: hdfs-3341.txt > > > After we commit HADOOP-8340, tests will fail because the minimum version is > configured to be 2.0.0/3.0.0 instead of the corresponding SNAPSHOT builds. > When we commit HADOOP-8340, we should update the minimums at the same time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3384) DataStreamer thread should be closed immediatly when failed to setup a PipelineForAppendOrRecovery
[
https://issues.apache.org/jira/browse/HDFS-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272033#comment-13272033
]
amith commented on HDFS-3384:
-
manually verified the patch, with testcase i was not able to assert the new
behaviour correctly
> DataStreamer thread should be closed immediatly when failed to setup a
> PipelineForAppendOrRecovery
> --
>
> Key: HDFS-3384
> URL: https://issues.apache.org/jira/browse/HDFS-3384
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 2.0.0
>Reporter: Brahma Reddy Battula
>Assignee: amith
> Attachments: HDFS-3384.patch
>
>
> Scenraio:
> =
> write a file
> corrupt block manually
> call append..
> {noformat}
> 2012-04-19 09:33:10,776 INFO hdfs.DFSClient
> (DFSOutputStream.java:createBlockOutputStream(1059)) - Exception in
> createBlockOutputStream
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1039)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:939)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> 2012-04-19 09:33:10,807 WARN hdfs.DFSClient (DFSOutputStream.java:run(549))
> - DataStreamer Exception
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:510)
> 2012-04-19 09:33:10,807 WARN hdfs.DFSClient
> (DFSOutputStream.java:hflush(1511)) - Error while syncing
> java.io.IOException: All datanodes 10.18.40.20:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:908)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> java.io.IOException: All datanodes 10.18.40.20:50010 are bad. Aborting...
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:908)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3385) ClassCastException when trying to append a file
[
https://issues.apache.org/jira/browse/HDFS-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-3385:
-
Attachment: h3385_20120509.patch
It turns out that the ClassCastException can be reproduced by the following.
# Create a new file with a full block.
# Append but not add any data so that the last block remains full and the last
block in INodeFileUnderConstruction is a BlockInfo but not
BlockInfoUnderConstruction.
# Append again.
h3385_20120509.patch:
- adds the test above;
- fixes the indentation.
> ClassCastException when trying to append a file
> ---
>
> Key: HDFS-3385
> URL: https://issues.apache.org/jira/browse/HDFS-3385
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Environment: HDFS
>Reporter: Brahma Reddy Battula
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h3385_20120508.patch, h3385_20120509.patch
>
>
> When I try to append a file I got
> {noformat}
> 2012-05-08 18:13:40,506 WARN util.KerberosName
> (KerberosName.java:(87)) - Kerberos krb5 configuration not found,
> setting default realm to empty
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo cannot be cast to
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoUnderConstruction
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1787)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1584)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1824)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:425)
> ...
> at org.apache.hadoop.hdfs.DFSClient.callAppend(DFSClient.java:1150)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1189)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1177)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:221)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:1)
> at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:981)
> at
> org.apache.hadoop.hdfs.server.datanode.DeleteMe.main(DeleteMe.java:26)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3385) ClassCastException when trying to append a file
[
https://issues.apache.org/jira/browse/HDFS-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-3385:
-
Status: Patch Available (was: Open)
> ClassCastException when trying to append a file
> ---
>
> Key: HDFS-3385
> URL: https://issues.apache.org/jira/browse/HDFS-3385
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Environment: HDFS
>Reporter: Brahma Reddy Battula
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h3385_20120508.patch, h3385_20120509.patch
>
>
> When I try to append a file I got
> {noformat}
> 2012-05-08 18:13:40,506 WARN util.KerberosName
> (KerberosName.java:(87)) - Kerberos krb5 configuration not found,
> setting default realm to empty
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo cannot be cast to
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoUnderConstruction
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1787)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1584)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1824)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:425)
> ...
> at org.apache.hadoop.hdfs.DFSClient.callAppend(DFSClient.java:1150)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1189)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1177)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:221)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:1)
> at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:981)
> at
> org.apache.hadoop.hdfs.server.datanode.DeleteMe.main(DeleteMe.java:26)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3385) ClassCastException when trying to append a file
[
https://issues.apache.org/jira/browse/HDFS-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272065#comment-13272065
]
Hadoop QA commented on HDFS-3385:
-
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12526286/h3385_20120509.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test
files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/2399//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2399//console
This message is automatically generated.
> ClassCastException when trying to append a file
> ---
>
> Key: HDFS-3385
> URL: https://issues.apache.org/jira/browse/HDFS-3385
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Environment: HDFS
>Reporter: Brahma Reddy Battula
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h3385_20120508.patch, h3385_20120509.patch
>
>
> When I try to append a file I got
> {noformat}
> 2012-05-08 18:13:40,506 WARN util.KerberosName
> (KerberosName.java:(87)) - Kerberos krb5 configuration not found,
> setting default realm to empty
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo cannot be cast to
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoUnderConstruction
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1787)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1584)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1824)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:425)
> ...
> at org.apache.hadoop.hdfs.DFSClient.callAppend(DFSClient.java:1150)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1189)
> at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:1177)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:221)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:1)
> at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:981)
> at
> org.apache.hadoop.hdfs.server.datanode.DeleteMe.main(DeleteMe.java:26)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3328) NPE in DataNode.getIpcPort
[
https://issues.apache.org/jira/browse/HDFS-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272082#comment-13272082
]
Hudson commented on HDFS-3328:
--
Integrated in Hadoop-Hdfs-trunk-Commit #2292 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2292/])
HDFS-3328. NPE in DataNode.getIpcPort. Contributed by Eli Collins (Revision
1336480)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336480
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> NPE in DataNode.getIpcPort
> --
>
> Key: HDFS-3328
> URL: https://issues.apache.org/jira/browse/HDFS-3328
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 2.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Eli Collins
>Priority: Minor
> Attachments: hdfs-3328.txt
>
>
> While running the tests, I have seen this exceptions.Tests passed.
> Not sure this is a problem.
> {quote}
> 2012-04-26 23:15:51,763 WARN hdfs.DFSClient (DFSOutputStream.java:run(710))
> - DFSOutputStream ResponseProcessor exception for block
> BP-1372255573-49.249.124.17-1335462329685:blk_-843504080180201_1005
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:95)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:657)
> Exception in thread "DataXceiver for client /127.0.0.1:52323 [Cleaning up]"
> java.lang.NullPointerException
> at org.apache.hadoop.ipc.Server$Listener.getAddress(Server.java:669)
> at org.apache.hadoop.ipc.Server.getListenerAddress(Server.java:1988)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getIpcPort(DataNode.java:882)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getDisplayName(DataNode.java:863)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:171)
> at java.lang.Thread.run(Unknown Source){quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3393) Rollback on newer release must not be allowed if the layout version in the snapshot is for older release.
[ https://issues.apache.org/jira/browse/HDFS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272083#comment-13272083 ] Suresh Srinivas commented on HDFS-3393: --- Yes, the change needs to be ported to 1.x. I will keep this for porting the change. > Rollback on newer release must not be allowed if the layout version in the > snapshot is for older release. > - > > Key: HDFS-3393 > URL: https://issues.apache.org/jira/browse/HDFS-3393 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 1.0.0, 2.0.0, 3.0.0 >Reporter: Suresh Srinivas > > After upgrading to a new release, the procedure to rollback to older release > is to start the namenode in older release with -rollback option. > As reported in HADOOP-8371, the rollback was allowed on the newer release. > During rollback, namenode must allow rollback only if the layout version of > the namenode matches that of the layout version in the snapshot that the > namenode is rolling back to. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3328) NPE in DataNode.getIpcPort
[
https://issues.apache.org/jira/browse/HDFS-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272085#comment-13272085
]
Hudson commented on HDFS-3328:
--
Integrated in Hadoop-Common-trunk-Commit #2217 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2217/])
HDFS-3328. NPE in DataNode.getIpcPort. Contributed by Eli Collins (Revision
1336480)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336480
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> NPE in DataNode.getIpcPort
> --
>
> Key: HDFS-3328
> URL: https://issues.apache.org/jira/browse/HDFS-3328
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 2.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Eli Collins
>Priority: Minor
> Attachments: hdfs-3328.txt
>
>
> While running the tests, I have seen this exceptions.Tests passed.
> Not sure this is a problem.
> {quote}
> 2012-04-26 23:15:51,763 WARN hdfs.DFSClient (DFSOutputStream.java:run(710))
> - DFSOutputStream ResponseProcessor exception for block
> BP-1372255573-49.249.124.17-1335462329685:blk_-843504080180201_1005
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:95)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:657)
> Exception in thread "DataXceiver for client /127.0.0.1:52323 [Cleaning up]"
> java.lang.NullPointerException
> at org.apache.hadoop.ipc.Server$Listener.getAddress(Server.java:669)
> at org.apache.hadoop.ipc.Server.getListenerAddress(Server.java:1988)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getIpcPort(DataNode.java:882)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getDisplayName(DataNode.java:863)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:171)
> at java.lang.Thread.run(Unknown Source){quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3395) NN doesn't start with HA+security enabled and HTTP address set to 0.0.0.0
Aaron T. Myers created HDFS-3395: Summary: NN doesn't start with HA+security enabled and HTTP address set to 0.0.0.0 Key: HDFS-3395 URL: https://issues.apache.org/jira/browse/HDFS-3395 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers DFSUtil#substituteForWildcardAddress subs in a default hostname if the given hostname is 0.0.0.0. However, this function throws an exception if the given hostname is set to 0.0.0.0 and security is enabled, regardless of whether the default hostname is also 0.0.0.0. This function shouldn't throw an exception unless both addresses are set to 0.0.0.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3393) Rollback on newer release must not be allowed if the layout version in the snapshot is for older release.
[ https://issues.apache.org/jira/browse/HDFS-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-3393: -- Affects Version/s: (was: 3.0.0) (was: 2.0.0) Setting the affected version to 1.0, since this problem is fixed in releases later than 0.22. > Rollback on newer release must not be allowed if the layout version in the > snapshot is for older release. > - > > Key: HDFS-3393 > URL: https://issues.apache.org/jira/browse/HDFS-3393 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 1.0.0 >Reporter: Suresh Srinivas > > After upgrading to a new release, the procedure to rollback to older release > is to start the namenode in older release with -rollback option. > As reported in HADOOP-8371, the rollback was allowed on the newer release. > During rollback, namenode must allow rollback only if the layout version of > the namenode matches that of the layout version in the snapshot that the > namenode is rolling back to. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3328) NPE in DataNode.getIpcPort
[
https://issues.apache.org/jira/browse/HDFS-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-3328:
--
Resolution: Fixed
Fix Version/s: 2.0.0
Target Version/s: (was: 2.0.0)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks guys. I've committed this to trunk and merged to branch-2 and
branch-2-alpha.
> NPE in DataNode.getIpcPort
> --
>
> Key: HDFS-3328
> URL: https://issues.apache.org/jira/browse/HDFS-3328
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 2.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Eli Collins
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: hdfs-3328.txt
>
>
> While running the tests, I have seen this exceptions.Tests passed.
> Not sure this is a problem.
> {quote}
> 2012-04-26 23:15:51,763 WARN hdfs.DFSClient (DFSOutputStream.java:run(710))
> - DFSOutputStream ResponseProcessor exception for block
> BP-1372255573-49.249.124.17-1335462329685:blk_-843504080180201_1005
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:95)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:657)
> Exception in thread "DataXceiver for client /127.0.0.1:52323 [Cleaning up]"
> java.lang.NullPointerException
> at org.apache.hadoop.ipc.Server$Listener.getAddress(Server.java:669)
> at org.apache.hadoop.ipc.Server.getListenerAddress(Server.java:1988)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getIpcPort(DataNode.java:882)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getDisplayName(DataNode.java:863)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:171)
> at java.lang.Thread.run(Unknown Source){quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3395) NN doesn't start with HA+security enabled and HTTP address set to 0.0.0.0
[ https://issues.apache.org/jira/browse/HDFS-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3395: - Attachment: HDFS-3395.patch Here's a patch which addresses the issue. No tests are included since security has to be enabled to test this code path. I tested it manually by setting the NN HTTP addresses to be wildcard in a secure HA setup, and confirming that checkpoints still proceed as expected. This patch also takes the liberty of fixing two typos I noticed in javadocs in the course of researching this bug. > NN doesn't start with HA+security enabled and HTTP address set to 0.0.0.0 > - > > Key: HDFS-3395 > URL: https://issues.apache.org/jira/browse/HDFS-3395 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3395.patch > > > DFSUtil#substituteForWildcardAddress subs in a default hostname if the given > hostname is 0.0.0.0. However, this function throws an exception if the given > hostname is set to 0.0.0.0 and security is enabled, regardless of whether the > default hostname is also 0.0.0.0. This function shouldn't throw an exception > unless both addresses are set to 0.0.0.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3395) NN doesn't start with HA+security enabled and HTTP address set to 0.0.0.0
[ https://issues.apache.org/jira/browse/HDFS-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3395: - Status: Patch Available (was: Open) > NN doesn't start with HA+security enabled and HTTP address set to 0.0.0.0 > - > > Key: HDFS-3395 > URL: https://issues.apache.org/jira/browse/HDFS-3395 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3395.patch > > > DFSUtil#substituteForWildcardAddress subs in a default hostname if the given > hostname is 0.0.0.0. However, this function throws an exception if the given > hostname is set to 0.0.0.0 and security is enabled, regardless of whether the > default hostname is also 0.0.0.0. This function shouldn't throw an exception > unless both addresses are set to 0.0.0.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3092) Enable journal protocol based editlog streaming for standby namenode
[ https://issues.apache.org/jira/browse/HDFS-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272101#comment-13272101 ] Eli Collins commented on HDFS-3092: --- Specifically, in HDFS-3092 we "write in parallel to all active/syncing journals. Writes must complete on all active journals or fail/timeout." therefore the latency of a transaction is that of the slowest JD. Note that the write must complete - a JD can't ack a write that's not on disk. In 3077 the latency is the slowest of the JDs in a quorum (ie if you have 5 JDs the latency is that of the 3rd fastest JD). > Enable journal protocol based editlog streaming for standby namenode > > > Key: HDFS-3092 > URL: https://issues.apache.org/jira/browse/HDFS-3092 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, name-node >Affects Versions: 0.24.0, 0.23.3 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: ComparisonofApproachesforHAJournals.pdf, JNStates.png, > MultipleSharedJournals.pdf, MultipleSharedJournals.pdf, > MultipleSharedJournals.pdf, Removingfilerdependency.pdf > > > Currently standby namenode relies on reading shared editlogs to stay current > with the active namenode, for namespace changes. BackupNode used streaming > edits from active namenode for doing the same. This jira is to explore using > journal protocol based editlog streams for the standby namenode. A daemon in > standby will get the editlogs from the active and write it to local edits. To > begin with, the existing standby mechanism of reading from a file, will > continue to be used, instead of from shared edits, from the local edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3328) NPE in DataNode.getIpcPort
[
https://issues.apache.org/jira/browse/HDFS-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272102#comment-13272102
]
Hudson commented on HDFS-3328:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #2234 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2234/])
HDFS-3328. NPE in DataNode.getIpcPort. Contributed by Eli Collins (Revision
1336480)
Result = ABORTED
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336480
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> NPE in DataNode.getIpcPort
> --
>
> Key: HDFS-3328
> URL: https://issues.apache.org/jira/browse/HDFS-3328
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 2.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Eli Collins
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: hdfs-3328.txt
>
>
> While running the tests, I have seen this exceptions.Tests passed.
> Not sure this is a problem.
> {quote}
> 2012-04-26 23:15:51,763 WARN hdfs.DFSClient (DFSOutputStream.java:run(710))
> - DFSOutputStream ResponseProcessor exception for block
> BP-1372255573-49.249.124.17-1335462329685:blk_-843504080180201_1005
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:95)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:657)
> Exception in thread "DataXceiver for client /127.0.0.1:52323 [Cleaning up]"
> java.lang.NullPointerException
> at org.apache.hadoop.ipc.Server$Listener.getAddress(Server.java:669)
> at org.apache.hadoop.ipc.Server.getListenerAddress(Server.java:1988)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getIpcPort(DataNode.java:882)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getDisplayName(DataNode.java:863)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:171)
> at java.lang.Thread.run(Unknown Source){quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2802) Support for RW/RO snapshots in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272103#comment-13272103 ] Eli Collins commented on HDFS-2802: --- Thanks Hari. Wrt scenario #3 doesn't this imply that the user would have to run two HBase instances on the same cluster - one running from /hbase and one running from /snapshot/hbase? If not then you need to shutdown the HBase instance running on the production data and then restart it with the snapshot. Maybe I'm missing something but both options (stopping the production instance and running two instances on the same production hardware) seem unrealistic. Also, don't HBase snapshots (HBASE-50) solve this problem? Are there other use cases aside from HBase? > Support for RW/RO snapshots in HDFS > --- > > Key: HDFS-2802 > URL: https://issues.apache.org/jira/browse/HDFS-2802 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node, name-node >Affects Versions: 0.24.0 >Reporter: Hari Mankude >Assignee: Hari Mankude > Attachments: snapshot-one-pager.pdf > > > Snapshots are point in time images of parts of the filesystem or the entire > filesystem. Snapshots can be a read-only or a read-write point in time copy > of the filesystem. There are several use cases for snapshots in HDFS. I will > post a detailed write-up soon with with more information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3395) NN doesn't start with HA+security enabled and HTTP address set to 0.0.0.0
[ https://issues.apache.org/jira/browse/HDFS-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272106#comment-13272106 ] Eli Collins commented on HDFS-3395: --- Looks good. +1 pending jenkins (please merge to branch-2-alpha as well) > NN doesn't start with HA+security enabled and HTTP address set to 0.0.0.0 > - > > Key: HDFS-3395 > URL: https://issues.apache.org/jira/browse/HDFS-3395 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3395.patch > > > DFSUtil#substituteForWildcardAddress subs in a default hostname if the given > hostname is 0.0.0.0. However, this function throws an exception if the given > hostname is set to 0.0.0.0 and security is enabled, regardless of whether the > default hostname is also 0.0.0.0. This function shouldn't throw an exception > unless both addresses are set to 0.0.0.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3396) FUSE build fails on Ubuntu 12.04
Colin Patrick McCabe created HDFS-3396:
--
Summary: FUSE build fails on Ubuntu 12.04
Key: HDFS-3396
URL: https://issues.apache.org/jira/browse/HDFS-3396
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
The HDFS FUSE-dfs build fails on Ubuntu 12.04 (and probably other OSes) with a
message like this:
{code}
/home/petru/work/ubeeko/hadoo.apache.org/0.23/hadoop-common/hadoop-hdfs-project/hadoop-hdfs/src/contrib/fuse-dfs/src/fuse_dfs.c:27:
undefined reference to `fuse_get_context'
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3396) FUSE build fails on Ubuntu 12.04
[
https://issues.apache.org/jira/browse/HDFS-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272113#comment-13272113
]
Colin Patrick McCabe commented on HDFS-3396:
s/LIBADD/LDADD for my previous comment.
> FUSE build fails on Ubuntu 12.04
>
>
> Key: HDFS-3396
> URL: https://issues.apache.org/jira/browse/HDFS-3396
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-3396.001.patch
>
>
> The HDFS FUSE-dfs build fails on Ubuntu 12.04 (and probably other OSes) with
> a message like this:
> {code}
> /home/petru/work/ubeeko/hadoo.apache.org/0.23/hadoop-common/hadoop-hdfs-project/hadoop-hdfs/src/contrib/fuse-dfs/src/fuse_dfs.c:27:
> undefined reference to `fuse_get_context'
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3396) FUSE build fails on Ubuntu 12.04
[
https://issues.apache.org/jira/browse/HDFS-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-3396:
---
Attachment: HDFS-3396.001.patch
* use LIBADD rather than LDFLAGS for libraries
* fix issue where we were not checking for libhdfs.so in usr/local/lib64
> FUSE build fails on Ubuntu 12.04
>
>
> Key: HDFS-3396
> URL: https://issues.apache.org/jira/browse/HDFS-3396
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-3396.001.patch
>
>
> The HDFS FUSE-dfs build fails on Ubuntu 12.04 (and probably other OSes) with
> a message like this:
> {code}
> /home/petru/work/ubeeko/hadoo.apache.org/0.23/hadoop-common/hadoop-hdfs-project/hadoop-hdfs/src/contrib/fuse-dfs/src/fuse_dfs.c:27:
> undefined reference to `fuse_get_context'
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3396) FUSE build fails on Ubuntu 12.04
[
https://issues.apache.org/jira/browse/HDFS-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-3396:
---
Status: Patch Available (was: Open)
> FUSE build fails on Ubuntu 12.04
>
>
> Key: HDFS-3396
> URL: https://issues.apache.org/jira/browse/HDFS-3396
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-3396.001.patch
>
>
> The HDFS FUSE-dfs build fails on Ubuntu 12.04 (and probably other OSes) with
> a message like this:
> {code}
> /home/petru/work/ubeeko/hadoo.apache.org/0.23/hadoop-common/hadoop-hdfs-project/hadoop-hdfs/src/contrib/fuse-dfs/src/fuse_dfs.c:27:
> undefined reference to `fuse_get_context'
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3397) Fuse-dfs won't compile on Ubuntu 12.04 due to improper automake variables
Andrew Bayer created HDFS-3397: -- Summary: Fuse-dfs won't compile on Ubuntu 12.04 due to improper automake variables Key: HDFS-3397 URL: https://issues.apache.org/jira/browse/HDFS-3397 Project: Hadoop HDFS Issue Type: Bug Components: contrib/fuse-dfs Affects Versions: 0.23.1 Reporter: Andrew Bayer See http://wiki.apache.org/hadoop/BuildFuseDfs023 - when building fuse-dfs on 11.10 or 12.04, the ordering of the linker command becomes significant, and the libraries ending up before the .o files causes the command to fail. http://ubuntuforums.org/archive/index.php/t-1858271.html suggests that this is because we're improperly using AM_LDFLAGS for libraries, when we should be using fuse_dfs_LDADD. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3397) Fuse-dfs won't compile on Ubuntu 12.04 due to improper automake variables
[ https://issues.apache.org/jira/browse/HDFS-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Bayer updated HDFS-3397: --- Attachment: HDFS-3397.patch Switches from AM_LDFLAGS to fuse_dfs_LDADD - linker command ordering is now correct and the command succeeds on Ubuntu 12.04. > Fuse-dfs won't compile on Ubuntu 12.04 due to improper automake variables > - > > Key: HDFS-3397 > URL: https://issues.apache.org/jira/browse/HDFS-3397 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Affects Versions: 0.23.1 >Reporter: Andrew Bayer > Attachments: HDFS-3397.patch > > > See http://wiki.apache.org/hadoop/BuildFuseDfs023 - when building fuse-dfs on > 11.10 or 12.04, the ordering of the linker command becomes significant, and > the libraries ending up before the .o files causes the command to fail. > http://ubuntuforums.org/archive/index.php/t-1858271.html suggests that this > is because we're improperly using AM_LDFLAGS for libraries, when we should be > using fuse_dfs_LDADD. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3396) FUSE build fails on Ubuntu 12.04
[
https://issues.apache.org/jira/browse/HDFS-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272125#comment-13272125
]
Hadoop QA commented on HDFS-3396:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12526304/HDFS-3396.001.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/2401//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2401//console
This message is automatically generated.
> FUSE build fails on Ubuntu 12.04
>
>
> Key: HDFS-3396
> URL: https://issues.apache.org/jira/browse/HDFS-3396
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-3396.001.patch
>
>
> The HDFS FUSE-dfs build fails on Ubuntu 12.04 (and probably other OSes) with
> a message like this:
> {code}
> /home/petru/work/ubeeko/hadoo.apache.org/0.23/hadoop-common/hadoop-hdfs-project/hadoop-hdfs/src/contrib/fuse-dfs/src/fuse_dfs.c:27:
> undefined reference to `fuse_get_context'
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3397) Fuse-dfs won't compile on Ubuntu 12.04 due to improper automake variables
[ https://issues.apache.org/jira/browse/HDFS-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins resolved HDFS-3397. --- Resolution: Duplicate Dupe of HDFS-3396 > Fuse-dfs won't compile on Ubuntu 12.04 due to improper automake variables > - > > Key: HDFS-3397 > URL: https://issues.apache.org/jira/browse/HDFS-3397 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Affects Versions: 0.23.1 >Reporter: Andrew Bayer > Attachments: HDFS-3397.patch > > > See http://wiki.apache.org/hadoop/BuildFuseDfs023 - when building fuse-dfs on > 11.10 or 12.04, the ordering of the linker command becomes significant, and > the libraries ending up before the .o files causes the command to fail. > http://ubuntuforums.org/archive/index.php/t-1858271.html suggests that this > is because we're improperly using AM_LDFLAGS for libraries, when we should be > using fuse_dfs_LDADD. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3396) FUSE build fails on Ubuntu 12.04
[
https://issues.apache.org/jira/browse/HDFS-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272128#comment-13272128
]
Eli Collins commented on HDFS-3396:
---
+1 tested on 10.10 as well. Just modifies the fuse native build so doesn't
need a test.
> FUSE build fails on Ubuntu 12.04
>
>
> Key: HDFS-3396
> URL: https://issues.apache.org/jira/browse/HDFS-3396
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-3396.001.patch
>
>
> The HDFS FUSE-dfs build fails on Ubuntu 12.04 (and probably other OSes) with
> a message like this:
> {code}
> /home/petru/work/ubeeko/hadoo.apache.org/0.23/hadoop-common/hadoop-hdfs-project/hadoop-hdfs/src/contrib/fuse-dfs/src/fuse_dfs.c:27:
> undefined reference to `fuse_get_context'
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3396) FUSE build fails on Ubuntu 12.04
[
https://issues.apache.org/jira/browse/HDFS-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-3396:
--
Resolution: Fixed
Fix Version/s: 2.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed and merged to branch-2 and branch-2-alpha. Thanks Colin!
> FUSE build fails on Ubuntu 12.04
>
>
> Key: HDFS-3396
> URL: https://issues.apache.org/jira/browse/HDFS-3396
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HDFS-3396.001.patch
>
>
> The HDFS FUSE-dfs build fails on Ubuntu 12.04 (and probably other OSes) with
> a message like this:
> {code}
> /home/petru/work/ubeeko/hadoo.apache.org/0.23/hadoop-common/hadoop-hdfs-project/hadoop-hdfs/src/contrib/fuse-dfs/src/fuse_dfs.c:27:
> undefined reference to `fuse_get_context'
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3396) FUSE build fails on Ubuntu 12.04
[
https://issues.apache.org/jira/browse/HDFS-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272131#comment-13272131
]
Hudson commented on HDFS-3396:
--
Integrated in Hadoop-Hdfs-trunk-Commit #2293 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2293/])
HDFS-3396. FUSE build fails on Ubuntu 12.04. Contributed by Colin Patrick
McCabe (Revision 1336495)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336495
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/fuse-dfs/src/Makefile.am
> FUSE build fails on Ubuntu 12.04
>
>
> Key: HDFS-3396
> URL: https://issues.apache.org/jira/browse/HDFS-3396
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HDFS-3396.001.patch
>
>
> The HDFS FUSE-dfs build fails on Ubuntu 12.04 (and probably other OSes) with
> a message like this:
> {code}
> /home/petru/work/ubeeko/hadoo.apache.org/0.23/hadoop-common/hadoop-hdfs-project/hadoop-hdfs/src/contrib/fuse-dfs/src/fuse_dfs.c:27:
> undefined reference to `fuse_get_context'
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3396) FUSE build fails on Ubuntu 12.04
[
https://issues.apache.org/jira/browse/HDFS-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272138#comment-13272138
]
Hudson commented on HDFS-3396:
--
Integrated in Hadoop-Common-trunk-Commit #2219 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2219/])
HDFS-3396. FUSE build fails on Ubuntu 12.04. Contributed by Colin Patrick
McCabe (Revision 1336495)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1336495
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/fuse-dfs/src/Makefile.am
> FUSE build fails on Ubuntu 12.04
>
>
> Key: HDFS-3396
> URL: https://issues.apache.org/jira/browse/HDFS-3396
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HDFS-3396.001.patch
>
>
> The HDFS FUSE-dfs build fails on Ubuntu 12.04 (and probably other OSes) with
> a message like this:
> {code}
> /home/petru/work/ubeeko/hadoo.apache.org/0.23/hadoop-common/hadoop-hdfs-project/hadoop-hdfs/src/contrib/fuse-dfs/src/fuse_dfs.c:27:
> undefined reference to `fuse_get_context'
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3395) NN doesn't start with HA+security enabled and HTTP address set to 0.0.0.0
[ https://issues.apache.org/jira/browse/HDFS-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272139#comment-13272139 ] Hadoop QA commented on HDFS-3395: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526302/HDFS-3395.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2400//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2400//console This message is automatically generated. > NN doesn't start with HA+security enabled and HTTP address set to 0.0.0.0 > - > > Key: HDFS-3395 > URL: https://issues.apache.org/jira/browse/HDFS-3395 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-3395.patch > > > DFSUtil#substituteForWildcardAddress subs in a default hostname if the given > hostname is 0.0.0.0. However, this function throws an exception if the given > hostname is set to 0.0.0.0 and security is enabled, regardless of whether the > default hostname is also 0.0.0.0. This function shouldn't throw an exception > unless both addresses are set to 0.0.0.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
