[jira] [Commented] (HDFS-2864) Remove redundant methods and a constant from FSDataset
[ https://issues.apache.org/jira/browse/HDFS-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197660#comment-13197660 ] Hudson commented on HDFS-2864: -- Integrated in Hadoop-Common-trunk-Commit #1630 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1630/]) HDFS-2864. Remove some redundant methods and the constant METADATA_VERSION from FSDataset. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238969 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java > Remove redundant methods and a constant from FSDataset > -- > > Key: HDFS-2864 > URL: https://issues.apache.org/jira/browse/HDFS-2864 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.24.0, 0.23.1 > > Attachments: h2864_20120131.patch > > > - METADATA_VERSION is declared in both FSDataset and BlockMetadataHeader. > - In FSDataset, the methods findBlockFile(..), getBlockFile(..) and > getFile(..) are very similar. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2864) Remove redundant methods and a constant from FSDataset
[ https://issues.apache.org/jira/browse/HDFS-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197657#comment-13197657 ] Hudson commented on HDFS-2864: -- Integrated in Hadoop-Hdfs-trunk-Commit #1701 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1701/]) HDFS-2864. Remove some redundant methods and the constant METADATA_VERSION from FSDataset. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238969 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java > Remove redundant methods and a constant from FSDataset > -- > > Key: HDFS-2864 > URL: https://issues.apache.org/jira/browse/HDFS-2864 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.24.0, 0.23.1 > > Attachments: h2864_20120131.patch > > > - METADATA_VERSION is declared in both FSDataset and BlockMetadataHeader. > - In FSDataset, the methods findBlockFile(..), getBlockFile(..) and > getFile(..) are very similar. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2864) Remove redundant methods and a constant from FSDataset
[ https://issues.apache.org/jira/browse/HDFS-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197656#comment-13197656 ] Hudson commented on HDFS-2864: -- Integrated in Hadoop-Hdfs-0.23-Commit #448 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/448/]) svn merge -c 1238969 from trunk for HDFS-2864. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238972 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java > Remove redundant methods and a constant from FSDataset > -- > > Key: HDFS-2864 > URL: https://issues.apache.org/jira/browse/HDFS-2864 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.24.0, 0.23.1 > > Attachments: h2864_20120131.patch > > > - METADATA_VERSION is declared in both FSDataset and BlockMetadataHeader. > - In FSDataset, the methods findBlockFile(..), getBlockFile(..) and > getFile(..) are very similar. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2864) Remove redundant methods and a constant from FSDataset
[ https://issues.apache.org/jira/browse/HDFS-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2864: - Resolution: Fixed Fix Version/s: 0.23.1 0.24.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I have committed this. > Remove redundant methods and a constant from FSDataset > -- > > Key: HDFS-2864 > URL: https://issues.apache.org/jira/browse/HDFS-2864 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.24.0, 0.23.1 > > Attachments: h2864_20120131.patch > > > - METADATA_VERSION is declared in both FSDataset and BlockMetadataHeader. > - In FSDataset, the methods findBlockFile(..), getBlockFile(..) and > getFile(..) are very similar. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2861) HA: checkpointing should verify that the dfs.http.address has been configured to a non-loopback for peer NN
[ https://issues.apache.org/jira/browse/HDFS-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197605#comment-13197605 ] Todd Lipcon commented on HDFS-2861: --- I ran the tests before commit and realized this needs a smidge more work. Updated patch coming soon or tomorrow. > HA: checkpointing should verify that the dfs.http.address has been configured > to a non-loopback for peer NN > --- > > Key: HDFS-2861 > URL: https://issues.apache.org/jira/browse/HDFS-2861 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Attachments: hdfs-2861.txt > > > In an HA setup I was running for the past week, I just noticed that > checkpoints weren't getting properly uploaded, since the SBN was connecting > to http://0.0.0.0:50070/ rather than the correct dfs.http.address. So, it was > uploading checkpoints to itself instead of the peer. We should add sanity > checks during startup to ensure that the configuration is correct. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1603) Namenode gets sticky if one of namenode storage volumes disappears (removed, unmounted, etc.)
[
https://issues.apache.org/jira/browse/HDFS-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197601#comment-13197601
]
Harsh J commented on HDFS-1603:
---
bq. I've noticed a failure during an unlock call that occurs AFTER a SD has
been detected as a failed point. The unlock call went ahead and blocked via a
native call to the NFS lock daemon - and since the NFS server was down, it just
hung (odd that the timeout did not apply, probably an nfs lockd issue, but I do
not feel its OK to unlock after a directory has caused a processIOError call).
Disregard the above. It was cause of a lockd bug in an earlier release of
CentOS as I'd suspected.
For:
bq. ATM and I just brainstormed about this a little bit over some iced coffee.
Though on the surface it doesn't look too hard to implement timeouts on namedir
operations, it would actually have to be done in a lot of places (eg
mkdirs/move calls on storage directories, writing edits, saving images, etc).
Timing out some of these things isn't entirely straightforward, since the
underlying calls aren't interruptible.
Since the hang is in processIOError or a call like that that handles the
dir-errors, lets have a timeout here instead? Should solve the same issue?
Though if it was the case like above, a thread may hang forever.
> Namenode gets sticky if one of namenode storage volumes disappears (removed,
> unmounted, etc.)
> -
>
> Key: HDFS-1603
> URL: https://issues.apache.org/jira/browse/HDFS-1603
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.21.0
>Reporter: Konstantin Boudnik
>
> While investigating failures on HDFS-1602 it became apparent that once a
> namenode storage volume is pulled out NN becomes completely "sticky" until
> {{FSImage:processIOError: removing storage}} move the storage from the active
> set. During this time none of normal NN operations are possible (e.g.
> creating a directory on HDFS timeouts eventually).
> In case of NFS this can be workaround'd with soft,intr,timeo,retrans
> settings. However, a better handling of the situation is apparently possible
> and needs to be implemented.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2861) HA: checkpointing should verify that the dfs.http.address has been configured to a non-loopback for peer NN
[ https://issues.apache.org/jira/browse/HDFS-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197597#comment-13197597 ] Eli Collins commented on HDFS-2861: --- +1 Nit, would add an assert or comment that HTTP_ADDRESS_DEFAULT is 0.0.0.0 and replace 50070 w DFS_NAMENODE_HTTP_PORT_DEFAULT. > HA: checkpointing should verify that the dfs.http.address has been configured > to a non-loopback for peer NN > --- > > Key: HDFS-2861 > URL: https://issues.apache.org/jira/browse/HDFS-2861 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Attachments: hdfs-2861.txt > > > In an HA setup I was running for the past week, I just noticed that > checkpoints weren't getting properly uploaded, since the SBN was connecting > to http://0.0.0.0:50070/ rather than the correct dfs.http.address. So, it was > uploading checkpoints to itself instead of the peer. We should add sanity > checks during startup to ensure that the configuration is correct. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2742) HA: observed dataloss in replication stress test
[ https://issues.apache.org/jira/browse/HDFS-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins resolved HDFS-2742. --- Resolution: Fixed Hadoop Flags: Reviewed I've committed this. > HA: observed dataloss in replication stress test > > > Key: HDFS-2742 > URL: https://issues.apache.org/jira/browse/HDFS-2742 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Attachments: hdfs-2742.txt, hdfs-2742.txt, hdfs-2742.txt, > hdfs-2742.txt, hdfs-2742.txt, hdfs-2742.txt, hdfs-2742.txt, hdfs-2742.txt, > log-colorized.txt > > > The replication stress test case failed over the weekend since one of the > replicas went missing. Still diagnosing the issue, but it seems like the > chain of events was something like: > - a block report was generated on one of the nodes while the block was being > written - thus the block report listed the block as RBW > - when the standby replayed this queued message, it was replayed after the > file was marked complete. Thus it marked this replica as corrupt > - it asked the DN holding the corrupt replica to delete it. And, I think, > removed it from the block map at this time. > - That DN then did another block report before receiving the deletion. This > caused it to be re-added to the block map, since it was "FINALIZED" now. > - Replication was lowered on the file, and it counted the above replica as > non-corrupt, and asked for the other replicas to be deleted. > - All replicas were lost. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2742) HA: observed dataloss in replication stress test
[ https://issues.apache.org/jira/browse/HDFS-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197589#comment-13197589 ] Eli Collins commented on HDFS-2742: --- Todd, thanks for the detailed explanation! +1 to the latest patch. > HA: observed dataloss in replication stress test > > > Key: HDFS-2742 > URL: https://issues.apache.org/jira/browse/HDFS-2742 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Attachments: hdfs-2742.txt, hdfs-2742.txt, hdfs-2742.txt, > hdfs-2742.txt, hdfs-2742.txt, hdfs-2742.txt, hdfs-2742.txt, hdfs-2742.txt, > log-colorized.txt > > > The replication stress test case failed over the weekend since one of the > replicas went missing. Still diagnosing the issue, but it seems like the > chain of events was something like: > - a block report was generated on one of the nodes while the block was being > written - thus the block report listed the block as RBW > - when the standby replayed this queued message, it was replayed after the > file was marked complete. Thus it marked this replica as corrupt > - it asked the DN holding the corrupt replica to delete it. And, I think, > removed it from the block map at this time. > - That DN then did another block report before receiving the deletion. This > caused it to be re-added to the block map, since it was "FINALIZED" now. > - Replication was lowered on the file, and it counted the above replica as > non-corrupt, and asked for the other replicas to be deleted. > - All replicas were lost. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2866) Standby does not start up due to a gap in transaction id
[ https://issues.apache.org/jira/browse/HDFS-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197515#comment-13197515 ] Aaron T. Myers commented on HDFS-2866: -- bq. don't we need a shared copy of fsimage? Nope, and configuring it as such would be an error. bq. If it is kept in local dirs, there would be a problem with two NNs formatting unless something like metadata copy is done from local dirs of NN1 to NN2. The intention is that when bootstrapping an HA cluster, one must format one NN and then copy the initially-empty metadata to the other NN. This is similar to bootstrapping other HA systems with persistent data, e.g. MySQL. See this JIRA which aims to improve this bootstrapping process: HDFS-2731 bq. Also, working with local image of fsimage might make it difficult to catch issues if there is divergence. How so? The standby NN performs checkpoints and thus will periodically download fsimage files from the active NN. Also note that the edits and image files include the transaction IDs they contain. > Standby does not start up due to a gap in transaction id > > > Key: HDFS-2866 > URL: https://issues.apache.org/jira/browse/HDFS-2866 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Priority: Critical > > Standby notices a gap in the transaction id in the shared.edits directory. > The transactions in dfs.edits.dir does not seem to have the gap. The gap > happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2866) Standby does not start up due to a gap in transaction id
[ https://issues.apache.org/jira/browse/HDFS-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197511#comment-13197511 ] Hari Mankude commented on HDFS-2866: don't we need a shared copy of fsimage? If it is kept in local dirs, there would be a problem with two NNs formatting unless something like metadata copy is done from local dirs of NN1 to NN2. Also, working with local image of fsimage might make it difficult to catch issues if there is divergence. keeping it in shared will ensure that second NN does not have to format. > Standby does not start up due to a gap in transaction id > > > Key: HDFS-2866 > URL: https://issues.apache.org/jira/browse/HDFS-2866 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Priority: Critical > > Standby notices a gap in the transaction id in the shared.edits directory. > The transactions in dfs.edits.dir does not seem to have the gap. The gap > happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2866) Standby does not start up due to a gap in transaction id
[ https://issues.apache.org/jira/browse/HDFS-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197499#comment-13197499 ] Todd Lipcon commented on HDFS-2866: --- Assuming /homes/hortonha is NFS-mounted (makes sense knowing what I know about Y grid infrastructure), then you have both of the NNs sharing the same directory for fsimage, which is incorrect. Also, I wouldn't recommend putting the fsimage dir inside the edits dir (though I don't see a reason it wouldn't work, it's certainly nonstandard) > Standby does not start up due to a gap in transaction id > > > Key: HDFS-2866 > URL: https://issues.apache.org/jira/browse/HDFS-2866 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Priority: Critical > > Standby notices a gap in the transaction id in the shared.edits directory. > The transactions in dfs.edits.dir does not seem to have the gap. The gap > happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2865) Standby namenode gets a "cannot lock storage" exception during startup
[ https://issues.apache.org/jira/browse/HDFS-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197500#comment-13197500 ] Todd Lipcon commented on HDFS-2865: --- If you have both NNs pointing to the same name dir, you will definitely have a problem. They should not share a namedir, since they'll both try to acquire the lock. The fact that the lock got deleted after the first failure is indeed a bug. > Standby namenode gets a "cannot lock storage" exception during startup > -- > > Key: HDFS-2865 > URL: https://issues.apache.org/jira/browse/HDFS-2865 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Assignee: Hari Mankude > > Standby NN is restarted. This is a follow-on to hdfs-2863. In this setup, > dfs.edits.dir is different from dfs.shared.edits.dir. During startup, standby > NN fails to acquire lock on the dfs.edits.dir. If standby NN is restarted > again, it seems to work fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2866) Standby does not start up due to a gap in transaction id
[ https://issues.apache.org/jira/browse/HDFS-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197495#comment-13197495 ] Aaron T. Myers commented on HDFS-2866: -- bq. name.dir and shared.edits.dir are nfs mounted. And both NNs were configured to access both of these NFS-mounted directories? > Standby does not start up due to a gap in transaction id > > > Key: HDFS-2866 > URL: https://issues.apache.org/jira/browse/HDFS-2866 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Priority: Critical > > Standby notices a gap in the transaction id in the shared.edits directory. > The transactions in dfs.edits.dir does not seem to have the gap. The gap > happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2866) Standby does not start up due to a gap in transaction id
[ https://issues.apache.org/jira/browse/HDFS-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197494#comment-13197494 ] Hari Mankude commented on HDFS-2866: @Aaron, name.dir and shared.edits.dir are nfs mounted. Regarding second question, the logs are from the newly active NN. The newly active NN just took over from the previous NN which was killed. I have highlighted the logs from transitiontoActive. dfs.name.dir /homes/hortonha/namenode/fsimage dfs.namenode.shared.edits.dir /homes/hortonha/namenode > Standby does not start up due to a gap in transaction id > > > Key: HDFS-2866 > URL: https://issues.apache.org/jira/browse/HDFS-2866 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Priority: Critical > > Standby notices a gap in the transaction id in the shared.edits directory. > The transactions in dfs.edits.dir does not seem to have the gap. The gap > happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2865) Standby namenode gets a "cannot lock storage" exception during startup
[ https://issues.apache.org/jira/browse/HDFS-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197491#comment-13197491 ] Hari Mankude commented on HDFS-2865: Both these are nfs directories. Lock issue is with name.dir dfs.name.dir /homes/hortonha/namenode/fsimage dfs.namenode.shared.edits.dir /homes/hortonha/namenode > Standby namenode gets a "cannot lock storage" exception during startup > -- > > Key: HDFS-2865 > URL: https://issues.apache.org/jira/browse/HDFS-2865 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Assignee: Hari Mankude > > Standby NN is restarted. This is a follow-on to hdfs-2863. In this setup, > dfs.edits.dir is different from dfs.shared.edits.dir. During startup, standby > NN fails to acquire lock on the dfs.edits.dir. If standby NN is restarted > again, it seems to work fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2845) SBN should not allow browsing of the file system via web UI
[ https://issues.apache.org/jira/browse/HDFS-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved HDFS-2845. -- Resolution: Fixed Fix Version/s: HA branch (HDFS-1623) Hadoop Flags: Reviewed I've just committed this to the branch. Thanks a lot for the contribution and for addressing the feedback, Bikas. > SBN should not allow browsing of the file system via web UI > --- > > Key: HDFS-2845 > URL: https://issues.apache.org/jira/browse/HDFS-2845 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-2485.HDFS-1623.patch, HDFS-2485.HDFS-1623.patch, > HDFS-2485.HDFS-1623.patch, HDFS-2485.HDFS-1623.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2861) HA: checkpointing should verify that the dfs.http.address has been configured to a non-loopback for peer NN
[ https://issues.apache.org/jira/browse/HDFS-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2861: -- Attachment: hdfs-2861.txt Attached patch does the following: - cleaned up the error messages when configuration isn't properly set up - improved the code so that, if the dfs.http.address is set to 0.0.0.0, that it uses the hostname from the RPC address instead (to match the secondary namenode) - added a sanity check that the IP address stored in the member variable is not 0.0.0.0. > HA: checkpointing should verify that the dfs.http.address has been configured > to a non-loopback for peer NN > --- > > Key: HDFS-2861 > URL: https://issues.apache.org/jira/browse/HDFS-2861 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Attachments: hdfs-2861.txt > > > In an HA setup I was running for the past week, I just noticed that > checkpoints weren't getting properly uploaded, since the SBN was connecting > to http://0.0.0.0:50070/ rather than the correct dfs.http.address. So, it was > uploading checkpoints to itself instead of the peer. We should add sanity > checks during startup to ensure that the configuration is correct. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2865) Standby namenode gets a "cannot lock storage" exception during startup
[
https://issues.apache.org/jira/browse/HDFS-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197482#comment-13197482
]
Todd Lipcon commented on HDFS-2865:
---
Can you please include your configuration snippet for the various dirs?
The fact that it works on a second restart is probably in fact a bug - we used
to have some bug where, when the lock file prevented startup, we'd still
accidentally _remove_ the lock during a {{finally}} clause, which is clearly
incorrect. So after you've "successfully" started up, you may have two NNs
writing to the same dir and the world will explode.
> Standby namenode gets a "cannot lock storage" exception during startup
> --
>
> Key: HDFS-2865
> URL: https://issues.apache.org/jira/browse/HDFS-2865
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Hari Mankude
>Assignee: Hari Mankude
>
> Standby NN is restarted. This is a follow-on to hdfs-2863. In this setup,
> dfs.edits.dir is different from dfs.shared.edits.dir. During startup, standby
> NN fails to acquire lock on the dfs.edits.dir. If standby NN is restarted
> again, it seems to work fine.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2845) SBN should not allow browsing of the file system via web UI
[ https://issues.apache.org/jira/browse/HDFS-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197479#comment-13197479 ] Aaron T. Myers commented on HDFS-2845: -- +1, the latest patch looks good to me. I'll commit this in a moment. > SBN should not allow browsing of the file system via web UI > --- > > Key: HDFS-2845 > URL: https://issues.apache.org/jira/browse/HDFS-2845 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: HDFS-2485.HDFS-1623.patch, HDFS-2485.HDFS-1623.patch, > HDFS-2485.HDFS-1623.patch, HDFS-2485.HDFS-1623.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2866) Standby does not start up due to a gap in transaction id
[ https://issues.apache.org/jira/browse/HDFS-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197477#comment-13197477 ] Aaron T. Myers commented on HDFS-2866: -- Hari, I still think you and Todd are talking about related, but slightly different, scenarios. Can you confirm that both your NNs were configured to be able to access both of these directories? If so, I think that would explain the confusion here. Can you also confirm that the logs you posted in the first comment on this JIRA were all from the previously-standby NN? > Standby does not start up due to a gap in transaction id > > > Key: HDFS-2866 > URL: https://issues.apache.org/jira/browse/HDFS-2866 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Priority: Critical > > Standby notices a gap in the transaction id in the shared.edits directory. > The transactions in dfs.edits.dir does not seem to have the gap. The gap > happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2288) Replicas awaiting recovery should return a full visible length
[ https://issues.apache.org/jira/browse/HDFS-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197474#comment-13197474 ] Todd Lipcon commented on HDFS-2288: --- Sorry to have left this ticket idle for such a long time. We're circling back to finish getting HBase tests passing on 23/trunk and ran afoul of this again. To continue the prior discussion: bq. We also provide read consistency, i.e. if N bytes are successful read from one datanode, then the same N bytes are available from all datanodes in the pipeline so that the client can switch to other datanodes and continue reading. By my understanding of the code, we do not provide this guarantee. For example, consider: - Client writing to 3 DNs - The network link between DN1 and DN2 in the pipeline is severed. - DN2 is sending an "ack" for some bytes back to DN1, but gets stuck sending over the severed network link During this window of time before the pipeline has timed out, if a client connects, the "bytesAcked" counter on DN3 will be higher than the "bytesAcked" counter on DN1. So, if a client connects to DN3, and then reconnects to DN1, it will have fewer visible bytes. So, I would counter that the above is not quite the right guarantee. Let me look deeper into the HBase test to understand whether it's a case that could happen in practice. Perhaps the correct result is not to return a "0" visible length but rather to throw an exception, forcing the client to retry or bail. > Replicas awaiting recovery should return a full visible length > -- > > Key: HDFS-2288 > URL: https://issues.apache.org/jira/browse/HDFS-2288 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.24.0 > > Attachments: hdfs-2288.txt > > > Currently, if the client calls getReplicaVisibleLength for a RWR, it returns > a visible length of 0. This causes one of HBase's tests to fail, and I > believe it's incorrect behavior. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2866) Standby does not start up due to a gap in transaction id
[ https://issues.apache.org/jira/browse/HDFS-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197462#comment-13197462 ] Hari Mankude commented on HDFS-2866: Todd's description is correct when there is local edits. We would need to ensure that shared edit dirs is always first in the list of dirs that is being written. This ensures that if failover happens when one of the updates is done and the other update is pending, we start off with the higher txid and this higher txid is available to the standby. However, the other issue is to how to get the local dirs back in sync with shared dir now that they have diverged. > Standby does not start up due to a gap in transaction id > > > Key: HDFS-2866 > URL: https://issues.apache.org/jira/browse/HDFS-2866 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Priority: Critical > > Standby notices a gap in the transaction id in the shared.edits directory. > The transactions in dfs.edits.dir does not seem to have the gap. The gap > happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2292) HA: HTTP fail-over
[ https://issues.apache.org/jira/browse/HDFS-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197458#comment-13197458 ] Eli Collins commented on HDFS-2292: --- There are three separate cases to handle: # web UI # fsck # HTTP-based file systems (hftp, webhdfs, httpfs) Might make more sense to discuss each independently on it's own jira. > HA: HTTP fail-over > -- > > Key: HDFS-2292 > URL: https://issues.apache.org/jira/browse/HDFS-2292 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: HA branch (HDFS-1623) > > > HDFS-1973 aims to enable client fail-over for HDFS RPCs. This JIRA is to > enable HTTP client fail over where appropriate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2866) Standby does not start up due to a gap in transaction id
[ https://issues.apache.org/jira/browse/HDFS-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197457#comment-13197457 ] Aaron T. Myers commented on HDFS-2866: -- bq. One possibility I can imagine is that, if the NN writes a txn group to the local disk and fsyncs successfully, and then fails before writing to the shared storage, we could have this scenario. Not unless we had a failover and failback, though, right? That doesn't seem to be what Hari was describing here. > Standby does not start up due to a gap in transaction id > > > Key: HDFS-2866 > URL: https://issues.apache.org/jira/browse/HDFS-2866 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Priority: Critical > > Standby notices a gap in the transaction id in the shared.edits directory. > The transactions in dfs.edits.dir does not seem to have the gap. The gap > happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2866) Standby does not start up due to a gap in transaction id
[ https://issues.apache.org/jira/browse/HDFS-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197446#comment-13197446 ] Todd Lipcon commented on HDFS-2866: --- One possibility I can imagine is that, if the NN writes a txn group to the local disk and fsyncs successfully, and then fails before writing to the shared storage, we could have this scenario. I think the solution is to make sure that the shared edits dirs always come first in the list of storage to write to. Does this sound like the issue you encountered? If not I'll move to a separate ticket. > Standby does not start up due to a gap in transaction id > > > Key: HDFS-2866 > URL: https://issues.apache.org/jira/browse/HDFS-2866 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Priority: Critical > > Standby notices a gap in the transaction id in the shared.edits directory. > The transactions in dfs.edits.dir does not seem to have the gap. The gap > happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2863) Failures observed if dfs.edits.dir and shared.edits.dir have same directories.
[ https://issues.apache.org/jira/browse/HDFS-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197396#comment-13197396 ] Bikas Saha commented on HDFS-2863: -- Looks like it does not since the BackupJournal does not seem to be linked to a URI. > Failures observed if dfs.edits.dir and shared.edits.dir have same directories. > -- > > Key: HDFS-2863 > URL: https://issues.apache.org/jira/browse/HDFS-2863 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Jitendra Nath Pandey >Assignee: Bikas Saha > > If same edits directory is configured in twice, both are treated > independently. Edit log roll is called on the same directory twice causing > exceptions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2863) Failures observed if dfs.edits.dir and shared.edits.dir have same directories.
[ https://issues.apache.org/jira/browse/HDFS-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197368#comment-13197368 ] Bikas Saha commented on HDFS-2863: -- Does it make sense to assert that the JournalSet must consists of distinct URI's as a policy decision. Then this check could be added to the JournalSet. > Failures observed if dfs.edits.dir and shared.edits.dir have same directories. > -- > > Key: HDFS-2863 > URL: https://issues.apache.org/jira/browse/HDFS-2863 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Jitendra Nath Pandey >Assignee: Bikas Saha > > If same edits directory is configured in twice, both are treated > independently. Edit log roll is called on the same directory twice causing > exceptions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2845) SBN should not allow browsing of the file system via web UI
[ https://issues.apache.org/jira/browse/HDFS-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated HDFS-2845: - Attachment: HDFS-2485.HDFS-1623.patch Sorry about that. I missed Jitendra's comments in Gmails conversation view. Adding the license in this patch. > SBN should not allow browsing of the file system via web UI > --- > > Key: HDFS-2845 > URL: https://issues.apache.org/jira/browse/HDFS-2845 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: HDFS-2485.HDFS-1623.patch, HDFS-2485.HDFS-1623.patch, > HDFS-2485.HDFS-1623.patch, HDFS-2485.HDFS-1623.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2867) org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-2867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated HDFS-2867:
--
Attachment:
TEST-org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations.xml
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations.txt
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations-output.txt
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
> fails intermittently
> -
>
> Key: HDFS-2867
> URL: https://issues.apache.org/jira/browse/HDFS-2867
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.24.0
>Reporter: Robert Joseph Evans
> Attachments:
> TEST-org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations.xml,
>
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations-output.txt,
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations.txt
>
>
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
> fails intermittently. It appears that it is caused by a previous test, or a
> previous part of this test not shutting down properly.
> {noformat}
> Tests run: 4, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 3.994 sec <<<
> FAILURE!
> test2NNRegistration(org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations)
> Time elapsed: 1.775 sec <<< ERROR!
> java.net.BindException: Problem binding to [localhost.localdomain:9928]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:675)
> at org.apache.hadoop.ipc.Server.bind(Server.java:306)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:393)
> at org.apache.hadoop.ipc.Server.(Server.java:1733)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:830)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2867) org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations fails intermittently
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations fails
intermittently
-
Key: HDFS-2867
URL: https://issues.apache.org/jira/browse/HDFS-2867
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.24.0
Reporter: Robert Joseph Evans
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations fails
intermittently. It appears that it is caused by a previous test, or a previous
part of this test not shutting down properly.
{noformat}
Tests run: 4, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 3.994 sec <<<
FAILURE!
test2NNRegistration(org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations)
Time elapsed: 1.775 sec <<< ERROR!
java.net.BindException: Problem binding to [localhost.localdomain:9928]
java.net.BindException: Address already in use; For more details see:
http://wiki.apache.org/hadoop/BindException
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:675)
at org.apache.hadoop.ipc.Server.bind(Server.java:306)
at org.apache.hadoop.ipc.Server$Listener.(Server.java:393)
at org.apache.hadoop.ipc.Server.(Server.java:1733)
at org.apache.hadoop.ipc.RPC$Server.(RPC.java:830)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2865) Standby namenode gets a "cannot lock storage" exception during startup
[ https://issues.apache.org/jira/browse/HDFS-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2865: - Component/s: name-node > Standby namenode gets a "cannot lock storage" exception during startup > -- > > Key: HDFS-2865 > URL: https://issues.apache.org/jira/browse/HDFS-2865 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Assignee: Hari Mankude > > Standby NN is restarted. This is a follow-on to hdfs-2863. In this setup, > dfs.edits.dir is different from dfs.shared.edits.dir. During startup, standby > NN fails to acquire lock on the dfs.edits.dir. If standby NN is restarted > again, it seems to work fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2845) SBN should not allow browsing of the file system via web UI
[ https://issues.apache.org/jira/browse/HDFS-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197346#comment-13197346 ] Aaron T. Myers commented on HDFS-2845: -- Hey Bikas, I think you missed adding the Apache license header per Jitendra's comment. Other than that the patch looks good, +1. > SBN should not allow browsing of the file system via web UI > --- > > Key: HDFS-2845 > URL: https://issues.apache.org/jira/browse/HDFS-2845 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: HDFS-2485.HDFS-1623.patch, HDFS-2485.HDFS-1623.patch, > HDFS-2485.HDFS-1623.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2845) SBN should not allow browsing of the file system via web UI
[ https://issues.apache.org/jira/browse/HDFS-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated HDFS-2845: - Attachment: HDFS-2485.HDFS-1623.patch New patch for review comments > SBN should not allow browsing of the file system via web UI > --- > > Key: HDFS-2845 > URL: https://issues.apache.org/jira/browse/HDFS-2845 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: HDFS-2485.HDFS-1623.patch, HDFS-2485.HDFS-1623.patch, > HDFS-2485.HDFS-1623.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2859) LOCAL_ADDRESS_MATCHER.match has NPE when called from DFSUtil.getSuffixIDs when the host is incorrect
[ https://issues.apache.org/jira/browse/HDFS-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197339#comment-13197339 ] Bikas Saha commented on HDFS-2859: -- Based on the response to my question above I may resolve [HDFS-2858|https://issues.apache.org/jira/browse/HDFS-2858] with this patch. If its ok to have erroneous hosts then it might work to simply log the exception and proceed. > LOCAL_ADDRESS_MATCHER.match has NPE when called from DFSUtil.getSuffixIDs > when the host is incorrect > > > Key: HDFS-2859 > URL: https://issues.apache.org/jira/browse/HDFS-2859 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Bikas Saha >Assignee: Bikas Saha >Priority: Minor > Attachments: HDFS-2859.HDFS-1623.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2859) LOCAL_ADDRESS_MATCHER.match has NPE when called from DFSUtil.getSuffixIDs when the host is incorrect
[ https://issues.apache.org/jira/browse/HDFS-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated HDFS-2859: - Attachment: HDFS-2859.HDFS-1623.patch Attaching a patch with a simple fix for the NPE. A general question I have is whether it is ok for certain name node hosts to be unresolved during startup. I guess it might be ok because some machines might legitimately be out of service and disconnected from the network. > LOCAL_ADDRESS_MATCHER.match has NPE when called from DFSUtil.getSuffixIDs > when the host is incorrect > > > Key: HDFS-2859 > URL: https://issues.apache.org/jira/browse/HDFS-2859 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Bikas Saha >Assignee: Bikas Saha >Priority: Minor > Attachments: HDFS-2859.HDFS-1623.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2866) Standby does not start up due to a gap in transaction id
[ https://issues.apache.org/jira/browse/HDFS-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197330#comment-13197330 ] Aaron T. Myers commented on HDFS-2866: -- bq. Here, I have two nfs-mounted edits directory(shared and one for fsimage/edits). During failover, edit logs in shared directory is missing a edit log entry while the edit log entry appears on the other edits directory. Basically, the two sets of edit logs are off by one txid. The new primary starts off with the highest txid. However, the standby notices a gap in the edit logs and cannot proceed further since the standby is using shared edits to roll forward. I don't follow how this state could occur. Could you perhaps write a test case that demonstrates this issue with a minicluster? > Standby does not start up due to a gap in transaction id > > > Key: HDFS-2866 > URL: https://issues.apache.org/jira/browse/HDFS-2866 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Priority: Critical > > Standby notices a gap in the transaction id in the shared.edits directory. > The transactions in dfs.edits.dir does not seem to have the gap. The gap > happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2866) Standby does not start up due to a gap in transaction id
[ https://issues.apache.org/jira/browse/HDFS-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197317#comment-13197317 ] Hari Mankude commented on HDFS-2866: Hi Aaron, I don't think this is same as hdfs-2794. Here, I have two nfs-mounted edits directory(shared and one for fsimage/edits). During failover, edit logs in shared directory is missing a edit log entry while the edit log entry appears on the other edits directory. Basically, the two sets of edit logs are off by one txid. The new primary starts off with the highest txid. However, the standby notices a gap in the edit logs and cannot proceed further since the standby is using shared edits to roll forward. > Standby does not start up due to a gap in transaction id > > > Key: HDFS-2866 > URL: https://issues.apache.org/jira/browse/HDFS-2866 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Priority: Critical > > Standby notices a gap in the transaction id in the shared.edits directory. > The transactions in dfs.edits.dir does not seem to have the gap. The gap > happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2866) Standby does not start up due to a gap in transaction id
[ https://issues.apache.org/jira/browse/HDFS-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197318#comment-13197318 ] Hari Mankude commented on HDFS-2866: version 0.24.0-SNAPSHOT, 1237534 > Standby does not start up due to a gap in transaction id > > > Key: HDFS-2866 > URL: https://issues.apache.org/jira/browse/HDFS-2866 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Priority: Critical > > Standby notices a gap in the transaction id in the shared.edits directory. > The transactions in dfs.edits.dir does not seem to have the gap. The gap > happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2865) Standby namenode gets a "cannot lock storage" exception during startup
[ https://issues.apache.org/jira/browse/HDFS-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2865: - Issue Type: Sub-task (was: Bug) Parent: HDFS-1623 > Standby namenode gets a "cannot lock storage" exception during startup > -- > > Key: HDFS-2865 > URL: https://issues.apache.org/jira/browse/HDFS-2865 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Assignee: Hari Mankude > > Standby NN is restarted. This is a follow-on to hdfs-2863. In this setup, > dfs.edits.dir is different from dfs.shared.edits.dir. During startup, standby > NN fails to acquire lock on the dfs.edits.dir. If standby NN is restarted > again, it seems to work fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2866) Standby does not start up due to a gap in transaction id
[ https://issues.apache.org/jira/browse/HDFS-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197308#comment-13197308 ] Aaron T. Myers commented on HDFS-2866: -- Hey Hari, this sounds like it might be a duplicate of HDFS-2794. Do you agree? Also, can you comment on what SVN revision you were testing this with? > Standby does not start up due to a gap in transaction id > > > Key: HDFS-2866 > URL: https://issues.apache.org/jira/browse/HDFS-2866 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Priority: Critical > > Standby notices a gap in the transaction id in the shared.edits directory. > The transactions in dfs.edits.dir does not seem to have the gap. The gap > happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2866) Standby does not start up due to a gap in transaction id
[ https://issues.apache.org/jira/browse/HDFS-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2866: - Issue Type: Sub-task (was: Bug) Parent: HDFS-1623 > Standby does not start up due to a gap in transaction id > > > Key: HDFS-2866 > URL: https://issues.apache.org/jira/browse/HDFS-2866 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Priority: Critical > > Standby notices a gap in the transaction id in the shared.edits directory. > The transactions in dfs.edits.dir does not seem to have the gap. The gap > happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2845) SBN should not allow browsing of the file system via web UI
[ https://issues.apache.org/jira/browse/HDFS-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197300#comment-13197300 ] Aaron T. Myers commented on HDFS-2845: -- Patch largely looks good. A few little comments: # No need for "@throws Exception" - doesn't help document anything. # I don't think there's any need to start a DN in this test, so might as well set numDataNodes to zero. Will speed up the test. > SBN should not allow browsing of the file system via web UI > --- > > Key: HDFS-2845 > URL: https://issues.apache.org/jira/browse/HDFS-2845 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: HDFS-2485.HDFS-1623.patch, HDFS-2485.HDFS-1623.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2835) Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue
[ https://issues.apache.org/jira/browse/HDFS-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197199#comment-13197199 ] Hudson commented on HDFS-2835: -- Integrated in Hadoop-Mapreduce-0.23-Commit #469 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/469/]) HDFS-2835. Merging change 1238779 from trunk to 0.23 suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238781 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs > Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue > -- > > Key: HDFS-2835 > URL: https://issues.apache.org/jira/browse/HDFS-2835 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2835.txt, HDFS-2835.txt, HDFS-2835.txt > > > https://builds.apache.org/job/PreCommit-HDFS-Build/1804//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html > shows a findbugs warning. It is unrelated to the patch being tested, and > has shown up on a few other JIRAS as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2835) Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue
[ https://issues.apache.org/jira/browse/HDFS-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197200#comment-13197200 ] Hudson commented on HDFS-2835: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1643 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1643/]) HDFS-2835. Fix findbugs and javadoc issue with GetConf.java. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238779 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java > Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue > -- > > Key: HDFS-2835 > URL: https://issues.apache.org/jira/browse/HDFS-2835 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2835.txt, HDFS-2835.txt, HDFS-2835.txt > > > https://builds.apache.org/job/PreCommit-HDFS-Build/1804//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html > shows a findbugs warning. It is unrelated to the patch being tested, and > has shown up on a few other JIRAS as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2835) Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue
[ https://issues.apache.org/jira/browse/HDFS-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197183#comment-13197183 ] Hudson commented on HDFS-2835: -- Integrated in Hadoop-Common-0.23-Commit #455 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/455/]) HDFS-2835. Merging change 1238779 from trunk to 0.23 suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238781 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs > Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue > -- > > Key: HDFS-2835 > URL: https://issues.apache.org/jira/browse/HDFS-2835 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2835.txt, HDFS-2835.txt, HDFS-2835.txt > > > https://builds.apache.org/job/PreCommit-HDFS-Build/1804//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html > shows a findbugs warning. It is unrelated to the patch being tested, and > has shown up on a few other JIRAS as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2835) Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue
[ https://issues.apache.org/jira/browse/HDFS-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197181#comment-13197181 ] Hudson commented on HDFS-2835: -- Integrated in Hadoop-Hdfs-trunk-Commit #1698 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1698/]) HDFS-2835. Fix findbugs and javadoc issue with GetConf.java. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238779 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java > Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue > -- > > Key: HDFS-2835 > URL: https://issues.apache.org/jira/browse/HDFS-2835 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2835.txt, HDFS-2835.txt, HDFS-2835.txt > > > https://builds.apache.org/job/PreCommit-HDFS-Build/1804//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html > shows a findbugs warning. It is unrelated to the patch being tested, and > has shown up on a few other JIRAS as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2835) Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue
[ https://issues.apache.org/jira/browse/HDFS-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197180#comment-13197180 ] Hudson commented on HDFS-2835: -- Integrated in Hadoop-Common-trunk-Commit #1627 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1627/]) HDFS-2835. Fix findbugs and javadoc issue with GetConf.java. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238779 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java > Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue > -- > > Key: HDFS-2835 > URL: https://issues.apache.org/jira/browse/HDFS-2835 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2835.txt, HDFS-2835.txt, HDFS-2835.txt > > > https://builds.apache.org/job/PreCommit-HDFS-Build/1804//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html > shows a findbugs warning. It is unrelated to the patch being tested, and > has shown up on a few other JIRAS as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2835) Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue
[ https://issues.apache.org/jira/browse/HDFS-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197179#comment-13197179 ] Hudson commented on HDFS-2835: -- Integrated in Hadoop-Hdfs-0.23-Commit #445 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/445/]) HDFS-2835. Merging change 1238779 from trunk to 0.23 suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238781 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs > Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue > -- > > Key: HDFS-2835 > URL: https://issues.apache.org/jira/browse/HDFS-2835 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2835.txt, HDFS-2835.txt, HDFS-2835.txt > > > https://builds.apache.org/job/PreCommit-HDFS-Build/1804//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html > shows a findbugs warning. It is unrelated to the patch being tested, and > has shown up on a few other JIRAS as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2835) Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue
[ https://issues.apache.org/jira/browse/HDFS-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2835: -- Affects Version/s: 0.23.0 Fix Version/s: 0.23.1 0.24.0 I committed the patch to 0.23 as well. > Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue > -- > > Key: HDFS-2835 > URL: https://issues.apache.org/jira/browse/HDFS-2835 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.24.0, 0.23.1 > > Attachments: HDFS-2835.txt, HDFS-2835.txt, HDFS-2835.txt > > > https://builds.apache.org/job/PreCommit-HDFS-Build/1804//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html > shows a findbugs warning. It is unrelated to the patch being tested, and > has shown up on a few other JIRAS as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2866) Standby does not start up due to a gap in transaction id
[ https://issues.apache.org/jira/browse/HDFS-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197171#comment-13197171 ] Hari Mankude commented on HDFS-2866: 2012-01-31 19:16:15,857 WARN ha.EditLogTailer (EditLogTailer.java:run(313)) - Edit log tailer interrupted java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:311) ---> Failover happens and this NN is the active NN 2012-01-31 19:16:15,858 INFO namenode.FSNamesystem (FSNamesystem.java:startActiveServices(515)) - Starting services required for active state 2012-01-31 19:16:15,860 INFO namenode.FileJournalManager (FileJournalManager.java:recoverUnfinalizedSegments(282)) - Recovering unfinalized segments in /homes/hortonha/namenode/fsimage/current 2012-01-31 19:16:15,881 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(94)) - Finalizing edits file /homes/hortonha/namenode/fsimage/current/edits_inprogress_0219665 -> /homes/hortonha/namenode/fsimage/current/edits_0219665-0220761 --> 220761 is the last txid in the dfs.edits.dir 2012-01-31 19:16:15,912 INFO namenode.FileJournalManager (FileJournalManager.java:recoverUnfinalizedSegments(282)) - Recovering unfinalized segments in /homes/hortonha/namenode/current 2012-01-31 19:16:15,931 INFO namenode.FileJournalManager (FileJournalManager.java:finalizeLogSegment(94)) - Finalizing edits file /homes/hortonha/namenode/current/edits_inprogress_0219665 -> /homes/hortonha/namenode/current/edits_0219665-0220760 ---> 220760 is the last txid in the shared.edits.dir 2012-01-31 19:16:15,933 INFO namenode.FSNamesystem (FSNamesystem.java:startActiveServices(527)) - Catching up to latest edits from old active before taking over writer role in edits logs. 2012-01-31 19:16:15,939 INFO namenode.FSImage (FSImage.java:loadEdits(682)) - Reading /homes/hortonha/namenode/fsimage/current/edits_0219665-0220761 expecting start txid #219665 2012-01-31 19:16:15,956 INFO namenode.FSImage (FSEditLogLoader.java:loadFSEdits(90)) - Edits file /homes/hortonha/namenode/fsimage/current/edits_0219665-0220761 of size 1048580 edits # 1097 loaded in 0 seconds. 2012-01-31 19:16:15,956 INFO blockmanagement.BlockManager (BlockManager.java:isReplicaCorrupt(1668)) - Received an RBW replica for block blk_-5497074126999415278_65205 on 98.137.233.237:50010: ignoring it, since the block is complete with the same generation stamp. 2012-01-31 19:16:16,465 INFO blockmanagement.BlockManager (BlockManager.java:processMisReplicatedBlocks(1932)) - Number of blocks being written= 3 2012-01-31 19:16:16,465 INFO namenode.FSNamesystem (FSNamesystem.java:startActiveServices(543)) - Will take over writing edit logs at txnid 220762 ---> takes over at 220762 resulting in a gap in edit logs in shared directory. Standby is stuck at this point. 2012-01-31 19:16:16,467 INFO namenode.FSEditLog (FSEditLog.java:startLogSegment(846)) - Starting log segment at 220762 > Standby does not start up due to a gap in transaction id > > > Key: HDFS-2866 > URL: https://issues.apache.org/jira/browse/HDFS-2866 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Priority: Critical > > Standby notices a gap in the transaction id in the shared.edits directory. > The transactions in dfs.edits.dir does not seem to have the gap. The gap > happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2835) Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue
[ https://issues.apache.org/jira/browse/HDFS-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2835: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed the patch to trunk. > Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue > -- > > Key: HDFS-2835 > URL: https://issues.apache.org/jira/browse/HDFS-2835 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Attachments: HDFS-2835.txt, HDFS-2835.txt, HDFS-2835.txt > > > https://builds.apache.org/job/PreCommit-HDFS-Build/1804//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html > shows a findbugs warning. It is unrelated to the patch being tested, and > has shown up on a few other JIRAS as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2866) Standby does not start up due to a gap in transaction id
Standby does not start up due to a gap in transaction id Key: HDFS-2866 URL: https://issues.apache.org/jira/browse/HDFS-2866 Project: Hadoop HDFS Issue Type: Bug Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Hari Mankude Priority: Critical Standby notices a gap in the transaction id in the shared.edits directory. The transactions in dfs.edits.dir does not seem to have the gap. The gap happens during a failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2845) SBN should not allow browsing of the file system via web UI
[ https://issues.apache.org/jira/browse/HDFS-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197166#comment-13197166 ] Jitendra Nath Pandey commented on HDFS-2845: Please add the apache license header to the new file. Apart from that the patch looks good. +1 > SBN should not allow browsing of the file system via web UI > --- > > Key: HDFS-2845 > URL: https://issues.apache.org/jira/browse/HDFS-2845 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: HDFS-2485.HDFS-1623.patch, HDFS-2485.HDFS-1623.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2865) Standby namenode gets a "cannot lock storage" exception during startup
[ https://issues.apache.org/jira/browse/HDFS-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Mankude updated HDFS-2865: --- Component/s: ha Affects Version/s: HA branch (HDFS-1623) > Standby namenode gets a "cannot lock storage" exception during startup > -- > > Key: HDFS-2865 > URL: https://issues.apache.org/jira/browse/HDFS-2865 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Hari Mankude >Assignee: Hari Mankude > > Standby NN is restarted. This is a follow-on to hdfs-2863. In this setup, > dfs.edits.dir is different from dfs.shared.edits.dir. During startup, standby > NN fails to acquire lock on the dfs.edits.dir. If standby NN is restarted > again, it seems to work fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2865) Standby namenode gets a "cannot lock storage" exception during startup
[ https://issues.apache.org/jira/browse/HDFS-2865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197161#comment-13197161 ] Hari Mankude commented on HDFS-2865: Logs from standby NN, 2012-01-31 19:16:23,883 INFO namenode.NameNode (FSDirectory.java:(142)) - Caching file names occuring more than 10 times 2012-01-31 19:16:23,904 INFO common.Storage (Storage.java:lock(585)) - Cannot lock storage /homes/hortonha/namenode/fsimage. The directory is already locked. 2012-01-31 19:16:23,905 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(199)) - Stopping NameNode metrics system... 2012-01-31 19:16:23,905 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(205)) - NameNode metrics system stopped. 2012-01-31 19:16:23,906 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(553)) - NameNode metrics system shutdown complete. 2012-01-31 19:16:23,906 ERROR namenode.NameNode (NameNode.java:main(898)) - Exception in namenode join java.io.IOException: Cannot lock storage /homes/hortonha/namenode/fsimage. The directory is already locked. at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:586) at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:435) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:263) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:178) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:441) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:380) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:351) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:385) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:540) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:526) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:836) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:894) 2012-01-31 19:16:23,914 INFO namenode.NameNode (StringUtils.java:run(605)) - SHUTDOWN_MSG: / SHUTDOWN_MSG: Shutting down NameNode at hrt11n27.cc1.ygridcore.net/98.137.234.163 / > Standby namenode gets a "cannot lock storage" exception during startup > -- > > Key: HDFS-2865 > URL: https://issues.apache.org/jira/browse/HDFS-2865 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Hari Mankude >Assignee: Hari Mankude > > Standby NN is restarted. This is a follow-on to hdfs-2863. In this setup, > dfs.edits.dir is different from dfs.shared.edits.dir. During startup, standby > NN fails to acquire lock on the dfs.edits.dir. If standby NN is restarted > again, it seems to work fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2865) Standby namenode gets a "cannot lock storage" exception during startup
Standby namenode gets a "cannot lock storage" exception during startup -- Key: HDFS-2865 URL: https://issues.apache.org/jira/browse/HDFS-2865 Project: Hadoop HDFS Issue Type: Bug Reporter: Hari Mankude Assignee: Hari Mankude Standby NN is restarted. This is a follow-on to hdfs-2863. In this setup, dfs.edits.dir is different from dfs.shared.edits.dir. During startup, standby NN fails to acquire lock on the dfs.edits.dir. If standby NN is restarted again, it seems to work fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2835) Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue
[ https://issues.apache.org/jira/browse/HDFS-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197151#comment-13197151 ] Hadoop QA commented on HDFS-2835: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512600/HDFS-2835.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFSInputChecker +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1829//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1829//console This message is automatically generated. > Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue > -- > > Key: HDFS-2835 > URL: https://issues.apache.org/jira/browse/HDFS-2835 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Attachments: HDFS-2835.txt, HDFS-2835.txt, HDFS-2835.txt > > > https://builds.apache.org/job/PreCommit-HDFS-Build/1804//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html > shows a findbugs warning. It is unrelated to the patch being tested, and > has shown up on a few other JIRAS as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2857) Cleanup BlockInfo class
[ https://issues.apache.org/jira/browse/HDFS-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197144#comment-13197144 ] Hudson commented on HDFS-2857: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1642 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1642/]) HDFS-2857. Cleanup BlockInfo class. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238747 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java > Cleanup BlockInfo class > --- > > Key: HDFS-2857 > URL: https://issues.apache.org/jira/browse/HDFS-2857 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0, 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2857.23.txt, HDFS-2857.txt > > > Following are some of the cleanup required: > # Remove unnecessary methods > # Add interface annotation > # Make some of the method private -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2857) Cleanup BlockInfo class
[ https://issues.apache.org/jira/browse/HDFS-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2857: -- Affects Version/s: 0.23.0 > Cleanup BlockInfo class > --- > > Key: HDFS-2857 > URL: https://issues.apache.org/jira/browse/HDFS-2857 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0, 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2857.23.txt, HDFS-2857.txt > > > Following are some of the cleanup required: > # Remove unnecessary methods > # Add interface annotation > # Make some of the method private -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2857) Cleanup BlockInfo class
[ https://issues.apache.org/jira/browse/HDFS-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2857: -- Attachment: HDFS-2857.23.txt 0.23 version of the patch, given merge from trunk cannot happen cleanly. > Cleanup BlockInfo class > --- > > Key: HDFS-2857 > URL: https://issues.apache.org/jira/browse/HDFS-2857 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2857.23.txt, HDFS-2857.txt > > > Following are some of the cleanup required: > # Remove unnecessary methods > # Add interface annotation > # Make some of the method private -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2857) Cleanup BlockInfo class
[ https://issues.apache.org/jira/browse/HDFS-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197129#comment-13197129 ] Hudson commented on HDFS-2857: -- Integrated in Hadoop-Common-trunk-Commit #1626 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1626/]) HDFS-2857. Cleanup BlockInfo class. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238747 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java > Cleanup BlockInfo class > --- > > Key: HDFS-2857 > URL: https://issues.apache.org/jira/browse/HDFS-2857 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2857.txt > > > Following are some of the cleanup required: > # Remove unnecessary methods > # Add interface annotation > # Make some of the method private -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2857) Cleanup BlockInfo class
[ https://issues.apache.org/jira/browse/HDFS-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197128#comment-13197128 ] Hudson commented on HDFS-2857: -- Integrated in Hadoop-Hdfs-trunk-Commit #1697 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1697/]) HDFS-2857. Cleanup BlockInfo class. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238747 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java > Cleanup BlockInfo class > --- > > Key: HDFS-2857 > URL: https://issues.apache.org/jira/browse/HDFS-2857 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2857.txt > > > Following are some of the cleanup required: > # Remove unnecessary methods > # Add interface annotation > # Make some of the method private -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2857) Cleanup BlockInfo class
[ https://issues.apache.org/jira/browse/HDFS-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197116#comment-13197116 ] Suresh Srinivas commented on HDFS-2857: --- Will do. > Cleanup BlockInfo class > --- > > Key: HDFS-2857 > URL: https://issues.apache.org/jira/browse/HDFS-2857 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2857.txt > > > Following are some of the cleanup required: > # Remove unnecessary methods > # Add interface annotation > # Make some of the method private -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2859) LOCAL_ADDRESS_MATCHER.match has NPE when called from DFSUtil.getSuffixIDs when the host is incorrect
[ https://issues.apache.org/jira/browse/HDFS-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2859: - Component/s: name-node ha Target Version/s: HA branch (HDFS-1623) Affects Version/s: HA branch (HDFS-1623) > LOCAL_ADDRESS_MATCHER.match has NPE when called from DFSUtil.getSuffixIDs > when the host is incorrect > > > Key: HDFS-2859 > URL: https://issues.apache.org/jira/browse/HDFS-2859 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Bikas Saha >Assignee: Bikas Saha >Priority: Minor > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2845) SBN should not allow browsing of the file system via web UI
[ https://issues.apache.org/jira/browse/HDFS-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2845: - Component/s: name-node ha Target Version/s: HA branch (HDFS-1623) Affects Version/s: HA branch (HDFS-1623) > SBN should not allow browsing of the file system via web UI > --- > > Key: HDFS-2845 > URL: https://issues.apache.org/jira/browse/HDFS-2845 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: HDFS-2485.HDFS-1623.patch, HDFS-2485.HDFS-1623.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2839) Nameservice id in file uri could cause issues
[ https://issues.apache.org/jira/browse/HDFS-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2839: - Component/s: name-node hdfs client ha Target Version/s: HA branch (HDFS-1623) > Nameservice id in file uri could cause issues > - > > Key: HDFS-2839 > URL: https://issues.apache.org/jira/browse/HDFS-2839 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, hdfs client, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > > There might be two issues with having a nameservice id in fully qualified > paths of the files. > 1) Some applications might be storing the fully qualified paths. They will > have hostnames from pre-ha installations, and they might break after the > failover. > 2) The fully qualified path from an ha cluster, won't be usable from a > different cluster that doesn't know about that particular namespace id. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2858) DFSUtil.getSuffixIDs silently ignores exception in NetUtils.createSocketAddr
[ https://issues.apache.org/jira/browse/HDFS-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2858: - Component/s: name-node ha Target Version/s: HA branch (HDFS-1623) Affects Version/s: HA branch (HDFS-1623) > DFSUtil.getSuffixIDs silently ignores exception in NetUtils.createSocketAddr > > > Key: HDFS-2858 > URL: https://issues.apache.org/jira/browse/HDFS-2858 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Bikas Saha >Assignee: Bikas Saha >Priority: Minor > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2864) Remove redundant methods and a constant from FSDataset
[ https://issues.apache.org/jira/browse/HDFS-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197114#comment-13197114 ] Tsz Wo (Nicholas), SZE commented on HDFS-2864: -- Thanks Suresh for the review. The failure of TestFSInputChecker and the findbugs warning are not related to this. > Remove redundant methods and a constant from FSDataset > -- > > Key: HDFS-2864 > URL: https://issues.apache.org/jira/browse/HDFS-2864 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h2864_20120131.patch > > > - METADATA_VERSION is declared in both FSDataset and BlockMetadataHeader. > - In FSDataset, the methods findBlockFile(..), getBlockFile(..) and > getFile(..) are very similar. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2863) Failures observed if dfs.edits.dir and shared.edits.dir have same directories.
[ https://issues.apache.org/jira/browse/HDFS-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2863: - Component/s: name-node ha Target Version/s: HA branch (HDFS-1623) Affects Version/s: HA branch (HDFS-1623) > Failures observed if dfs.edits.dir and shared.edits.dir have same directories. > -- > > Key: HDFS-2863 > URL: https://issues.apache.org/jira/browse/HDFS-2863 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Jitendra Nath Pandey >Assignee: Bikas Saha > > If same edits directory is configured in twice, both are treated > independently. Edit log roll is called on the same directory twice causing > exceptions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2861) HA: checkpointing should verify that the dfs.http.address has been configured to a non-loopback for peer NN
[ https://issues.apache.org/jira/browse/HDFS-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2861: -- Component/s: ha > HA: checkpointing should verify that the dfs.http.address has been configured > to a non-loopback for peer NN > --- > > Key: HDFS-2861 > URL: https://issues.apache.org/jira/browse/HDFS-2861 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > > In an HA setup I was running for the past week, I just noticed that > checkpoints weren't getting properly uploaded, since the SBN was connecting > to http://0.0.0.0:50070/ rather than the correct dfs.http.address. So, it was > uploading checkpoints to itself instead of the peer. We should add sanity > checks during startup to ensure that the configuration is correct. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2815) Namenode is not coming out of safemode when we perform ( NN crash + restart ) . Also FSCK report shows blocks missed.
[
https://issues.apache.org/jira/browse/HDFS-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197109#comment-13197109
]
Uma Maheswara Rao G commented on HDFS-2815:
---
{quote}
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9)
warnings.
{quote}
not related to this patch . It is handling by HDFS-2835
Test failures also unrelated to this patch.
> Namenode is not coming out of safemode when we perform ( NN crash + restart )
> . Also FSCK report shows blocks missed.
> --
>
> Key: HDFS-2815
> URL: https://issues.apache.org/jira/browse/HDFS-2815
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.22.0, 0.24.0, 0.23.1, 1.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Critical
> Attachments: HDFS-2815.patch
>
>
> When tested the HA(internal) with continuous switch with some 5mins gap,
> found some *blocks missed* and namenode went into safemode after next switch.
>
>After the analysis, i found that this files already deleted by clients.
> But i don't see any delete commands logs namenode log files. But namenode
> added that blocks to invalidateSets and DNs deleted the blocks.
>When restart of the namenode, it went into safemode and expecting some
> more blocks to come out of safemode.
>Here the reason could be that, file has been deleted in memory and added
> into invalidates after this it is trying to sync the edits into editlog file.
> By that time NN asked DNs to delete that blocks. Now namenode shuts down
> before persisting to editlogs.( log behind)
>Due to this reason, we may not get the INFO logs about delete, and when we
> restart the Namenode (in my scenario it is again switch), Namenode expects
> this deleted blocks also, as delete request is not persisted into editlog
> before.
>I reproduced this scenario with bedug points. *I feel, We should not add
> the blocks to invalidates before persisting into Editlog*.
> Note: for switch, we used kill -9 (force kill)
> I am currently in 0.20.2 version. Same verified in 0.23 as well in normal
> crash + restart scenario.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2864) Remove redundant methods and a constant from FSDataset
[ https://issues.apache.org/jira/browse/HDFS-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197104#comment-13197104 ] Hadoop QA commented on HDFS-2864: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512593/h2864_20120131.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 21 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFSInputChecker +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1828//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1828//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1828//console This message is automatically generated. > Remove redundant methods and a constant from FSDataset > -- > > Key: HDFS-2864 > URL: https://issues.apache.org/jira/browse/HDFS-2864 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h2864_20120131.patch > > > - METADATA_VERSION is declared in both FSDataset and BlockMetadataHeader. > - In FSDataset, the methods findBlockFile(..), getBlockFile(..) and > getFile(..) are very similar. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2827) Cannot save namespace after renaming a directory above a file with an open lease
[
https://issues.apache.org/jira/browse/HDFS-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197091#comment-13197091
]
Hudson commented on HDFS-2827:
--
Integrated in Hadoop-Mapreduce-0.23-Commit #467 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/467/])
svn merge -c 1238700 from trunk for HDFS-2827.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238709
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
> Cannot save namespace after renaming a directory above a file with an open
> lease
>
>
> Key: HDFS-2827
> URL: https://issues.apache.org/jira/browse/HDFS-2827
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.24.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Fix For: 0.24.0, 0.23.1
>
> Attachments: HDFS-2827-test.patch, HDFS-2827.patch
>
>
> When i execute the following operations and wait for checkpoint to complete.
> fs.mkdirs(new Path("/test1"));
> FSDataOutputStream create = fs.create(new Path("/test/abc.txt")); //dont close
> fs.rename(new Path("/test/"), new Path("/test1/"));
> Check-pointing is failing with the following exception.
> 2012-01-23 15:03:14,204 ERROR namenode.FSImage (FSImage.java:run(795)) -
> Unable to save image for
> E:\HDFS-1623\hadoop-hdfs-project\hadoop-hdfs\build\test\data\dfs\name3
> java.io.IOException: saveLeases found path /test1/est/abc.txt but no matching
> entry in namespace.[/test1/est/abc.txt]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4336)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Saver.save(FSImageFormat.java:588)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:761)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:789)
> at java.lang.Thread.run(Unknown Source)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2815) Namenode is not coming out of safemode when we perform ( NN crash + restart ) . Also FSCK report shows blocks missed.
[ https://issues.apache.org/jira/browse/HDFS-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197093#comment-13197093 ] Hadoop QA commented on HDFS-2815: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511806/HDFS-2815.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFSInputChecker +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1827//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1827//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1827//console This message is automatically generated. > Namenode is not coming out of safemode when we perform ( NN crash + restart ) > . Also FSCK report shows blocks missed. > -- > > Key: HDFS-2815 > URL: https://issues.apache.org/jira/browse/HDFS-2815 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0, 0.24.0, 0.23.1, 1.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Critical > Attachments: HDFS-2815.patch > > > When tested the HA(internal) with continuous switch with some 5mins gap, > found some *blocks missed* and namenode went into safemode after next switch. > >After the analysis, i found that this files already deleted by clients. > But i don't see any delete commands logs namenode log files. But namenode > added that blocks to invalidateSets and DNs deleted the blocks. >When restart of the namenode, it went into safemode and expecting some > more blocks to come out of safemode. >Here the reason could be that, file has been deleted in memory and added > into invalidates after this it is trying to sync the edits into editlog file. > By that time NN asked DNs to delete that blocks. Now namenode shuts down > before persisting to editlogs.( log behind) >Due to this reason, we may not get the INFO logs about delete, and when we > restart the Namenode (in my scenario it is again switch), Namenode expects > this deleted blocks also, as delete request is not persisted into editlog > before. >I reproduced this scenario with bedug points. *I feel, We should not add > the blocks to invalidates before persisting into Editlog*. > Note: for switch, we used kill -9 (force kill) > I am currently in 0.20.2 version. Same verified in 0.23 as well in normal > crash + restart scenario. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2857) Cleanup BlockInfo class
[ https://issues.apache.org/jira/browse/HDFS-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197090#comment-13197090 ] Eli Collins commented on HDFS-2857: --- +1 lgtm. Mind merging to 23? Would be good to have the annotations there as well. > Cleanup BlockInfo class > --- > > Key: HDFS-2857 > URL: https://issues.apache.org/jira/browse/HDFS-2857 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2857.txt > > > Following are some of the cleanup required: > # Remove unnecessary methods > # Add interface annotation > # Make some of the method private -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2857) Cleanup BlockInfo class
[ https://issues.apache.org/jira/browse/HDFS-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2857: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) This patch cleaned up BlockInfo class (see description). I did not add new tests. Committed the patch. > Cleanup BlockInfo class > --- > > Key: HDFS-2857 > URL: https://issues.apache.org/jira/browse/HDFS-2857 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2857.txt > > > Following are some of the cleanup required: > # Remove unnecessary methods > # Add interface annotation > # Make some of the method private -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2827) Cannot save namespace after renaming a directory above a file with an open lease
[
https://issues.apache.org/jira/browse/HDFS-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197085#comment-13197085
]
Hudson commented on HDFS-2827:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #1640 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1640/])
HDFS-2827. When the parent of a directory is the root, renaming the
directory results in leases updated incorrectly. Contributed by Uma Maheswara
Rao G
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238700
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
> Cannot save namespace after renaming a directory above a file with an open
> lease
>
>
> Key: HDFS-2827
> URL: https://issues.apache.org/jira/browse/HDFS-2827
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.24.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Fix For: 0.24.0, 0.23.1
>
> Attachments: HDFS-2827-test.patch, HDFS-2827.patch
>
>
> When i execute the following operations and wait for checkpoint to complete.
> fs.mkdirs(new Path("/test1"));
> FSDataOutputStream create = fs.create(new Path("/test/abc.txt")); //dont close
> fs.rename(new Path("/test/"), new Path("/test1/"));
> Check-pointing is failing with the following exception.
> 2012-01-23 15:03:14,204 ERROR namenode.FSImage (FSImage.java:run(795)) -
> Unable to save image for
> E:\HDFS-1623\hadoop-hdfs-project\hadoop-hdfs\build\test\data\dfs\name3
> java.io.IOException: saveLeases found path /test1/est/abc.txt but no matching
> entry in namespace.[/test1/est/abc.txt]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4336)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Saver.save(FSImageFormat.java:588)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:761)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:789)
> at java.lang.Thread.run(Unknown Source)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2857) Cleanup BlockInfo class
[ https://issues.apache.org/jira/browse/HDFS-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197080#comment-13197080 ] Suresh Srinivas commented on HDFS-2857: --- I did consider that, however, setPrevious is called without setNext. Also when list manipulation is being done, datanode might already be set. > Cleanup BlockInfo class > --- > > Key: HDFS-2857 > URL: https://issues.apache.org/jira/browse/HDFS-2857 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2857.txt > > > Following are some of the cleanup required: > # Remove unnecessary methods > # Add interface annotation > # Make some of the method private -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2857) Cleanup BlockInfo class
[ https://issues.apache.org/jira/browse/HDFS-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2857: -- Issue Type: Improvement (was: Bug) > Cleanup BlockInfo class > --- > > Key: HDFS-2857 > URL: https://issues.apache.org/jira/browse/HDFS-2857 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2857.txt > > > Following are some of the cleanup required: > # Remove unnecessary methods > # Add interface annotation > # Make some of the method private -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2864) Remove redundant methods and a constant from FSDataset
[ https://issues.apache.org/jira/browse/HDFS-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197072#comment-13197072 ] Suresh Srinivas commented on HDFS-2864: --- +1 for the patch. > Remove redundant methods and a constant from FSDataset > -- > > Key: HDFS-2864 > URL: https://issues.apache.org/jira/browse/HDFS-2864 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h2864_20120131.patch > > > - METADATA_VERSION is declared in both FSDataset and BlockMetadataHeader. > - In FSDataset, the methods findBlockFile(..), getBlockFile(..) and > getFile(..) are very similar. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2827) Cannot save namespace after renaming a directory above a file with an open lease
[
https://issues.apache.org/jira/browse/HDFS-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197062#comment-13197062
]
Tsz Wo (Nicholas), SZE commented on HDFS-2827:
--
Aaron, thanks for your understanding. :)
> Cannot save namespace after renaming a directory above a file with an open
> lease
>
>
> Key: HDFS-2827
> URL: https://issues.apache.org/jira/browse/HDFS-2827
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.24.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Fix For: 0.24.0, 0.23.1
>
> Attachments: HDFS-2827-test.patch, HDFS-2827.patch
>
>
> When i execute the following operations and wait for checkpoint to complete.
> fs.mkdirs(new Path("/test1"));
> FSDataOutputStream create = fs.create(new Path("/test/abc.txt")); //dont close
> fs.rename(new Path("/test/"), new Path("/test1/"));
> Check-pointing is failing with the following exception.
> 2012-01-23 15:03:14,204 ERROR namenode.FSImage (FSImage.java:run(795)) -
> Unable to save image for
> E:\HDFS-1623\hadoop-hdfs-project\hadoop-hdfs\build\test\data\dfs\name3
> java.io.IOException: saveLeases found path /test1/est/abc.txt but no matching
> entry in namespace.[/test1/est/abc.txt]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4336)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Saver.save(FSImageFormat.java:588)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:761)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:789)
> at java.lang.Thread.run(Unknown Source)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2827) Cannot save namespace after renaming a directory above a file with an open lease
[
https://issues.apache.org/jira/browse/HDFS-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197057#comment-13197057
]
Hudson commented on HDFS-2827:
--
Integrated in Hadoop-Hdfs-0.23-Commit #443 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/443/])
svn merge -c 1238700 from trunk for HDFS-2827.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238709
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
> Cannot save namespace after renaming a directory above a file with an open
> lease
>
>
> Key: HDFS-2827
> URL: https://issues.apache.org/jira/browse/HDFS-2827
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.24.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Fix For: 0.24.0, 0.23.1
>
> Attachments: HDFS-2827-test.patch, HDFS-2827.patch
>
>
> When i execute the following operations and wait for checkpoint to complete.
> fs.mkdirs(new Path("/test1"));
> FSDataOutputStream create = fs.create(new Path("/test/abc.txt")); //dont close
> fs.rename(new Path("/test/"), new Path("/test1/"));
> Check-pointing is failing with the following exception.
> 2012-01-23 15:03:14,204 ERROR namenode.FSImage (FSImage.java:run(795)) -
> Unable to save image for
> E:\HDFS-1623\hadoop-hdfs-project\hadoop-hdfs\build\test\data\dfs\name3
> java.io.IOException: saveLeases found path /test1/est/abc.txt but no matching
> entry in namespace.[/test1/est/abc.txt]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4336)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Saver.save(FSImageFormat.java:588)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:761)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:789)
> at java.lang.Thread.run(Unknown Source)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2827) Cannot save namespace after renaming a directory above a file with an open lease
[
https://issues.apache.org/jira/browse/HDFS-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197055#comment-13197055
]
Aaron T. Myers commented on HDFS-2827:
--
bq. Hi Aaron, sorry that I did not see you message earlier. You can still
review it and, if you find any problem, please feel free to revert it.
No biggie. I'll take a look now.
bq. BTW, the patch is available for more than a week and Uma has particularly
asked you to take a look ...
Yea, Uma also emailed me offline and I told him I would take a look at it, but
it wasn't my highest priority at the moment because of some deadlines I had
yesterday. Hence, I had intended to look today. I probably should've said that
on this JIRA. :)
> Cannot save namespace after renaming a directory above a file with an open
> lease
>
>
> Key: HDFS-2827
> URL: https://issues.apache.org/jira/browse/HDFS-2827
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.24.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Fix For: 0.24.0, 0.23.1
>
> Attachments: HDFS-2827-test.patch, HDFS-2827.patch
>
>
> When i execute the following operations and wait for checkpoint to complete.
> fs.mkdirs(new Path("/test1"));
> FSDataOutputStream create = fs.create(new Path("/test/abc.txt")); //dont close
> fs.rename(new Path("/test/"), new Path("/test1/"));
> Check-pointing is failing with the following exception.
> 2012-01-23 15:03:14,204 ERROR namenode.FSImage (FSImage.java:run(795)) -
> Unable to save image for
> E:\HDFS-1623\hadoop-hdfs-project\hadoop-hdfs\build\test\data\dfs\name3
> java.io.IOException: saveLeases found path /test1/est/abc.txt but no matching
> entry in namespace.[/test1/est/abc.txt]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4336)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Saver.save(FSImageFormat.java:588)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:761)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:789)
> at java.lang.Thread.run(Unknown Source)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2827) Cannot save namespace after renaming a directory above a file with an open lease
[
https://issues.apache.org/jira/browse/HDFS-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197051#comment-13197051
]
Hudson commented on HDFS-2827:
--
Integrated in Hadoop-Common-0.23-Commit #452 (See
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/452/])
svn merge -c 1238700 from trunk for HDFS-2827.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238709
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
> Cannot save namespace after renaming a directory above a file with an open
> lease
>
>
> Key: HDFS-2827
> URL: https://issues.apache.org/jira/browse/HDFS-2827
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.24.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Fix For: 0.24.0, 0.23.1
>
> Attachments: HDFS-2827-test.patch, HDFS-2827.patch
>
>
> When i execute the following operations and wait for checkpoint to complete.
> fs.mkdirs(new Path("/test1"));
> FSDataOutputStream create = fs.create(new Path("/test/abc.txt")); //dont close
> fs.rename(new Path("/test/"), new Path("/test1/"));
> Check-pointing is failing with the following exception.
> 2012-01-23 15:03:14,204 ERROR namenode.FSImage (FSImage.java:run(795)) -
> Unable to save image for
> E:\HDFS-1623\hadoop-hdfs-project\hadoop-hdfs\build\test\data\dfs\name3
> java.io.IOException: saveLeases found path /test1/est/abc.txt but no matching
> entry in namespace.[/test1/est/abc.txt]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4336)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Saver.save(FSImageFormat.java:588)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:761)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:789)
> at java.lang.Thread.run(Unknown Source)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2835) Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue
[ https://issues.apache.org/jira/browse/HDFS-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2835: -- Attachment: HDFS-2835.txt New patch with javadoc warnings fixes > Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue > -- > > Key: HDFS-2835 > URL: https://issues.apache.org/jira/browse/HDFS-2835 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Attachments: HDFS-2835.txt, HDFS-2835.txt, HDFS-2835.txt > > > https://builds.apache.org/job/PreCommit-HDFS-Build/1804//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html > shows a findbugs warning. It is unrelated to the patch being tested, and > has shown up on a few other JIRAS as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2827) Cannot save namespace after renaming a directory above a file with an open lease
[
https://issues.apache.org/jira/browse/HDFS-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-2827:
-
Resolution: Incomplete
Fix Version/s: 0.23.1
0.24.0
Status: Resolved (was: Patch Available)
I have committed the patch to trunk and 0.23. Thanks, Uma!
Hi Aaron, sorry that I did not see you message earlier. You can still review
it and, if you find any problem, please feel free to revert it.
BTW, the patch is available for more than a week and Uma has particularly asked
you to take a look ...
> Cannot save namespace after renaming a directory above a file with an open
> lease
>
>
> Key: HDFS-2827
> URL: https://issues.apache.org/jira/browse/HDFS-2827
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.24.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Fix For: 0.24.0, 0.23.1
>
> Attachments: HDFS-2827-test.patch, HDFS-2827.patch
>
>
> When i execute the following operations and wait for checkpoint to complete.
> fs.mkdirs(new Path("/test1"));
> FSDataOutputStream create = fs.create(new Path("/test/abc.txt")); //dont close
> fs.rename(new Path("/test/"), new Path("/test1/"));
> Check-pointing is failing with the following exception.
> 2012-01-23 15:03:14,204 ERROR namenode.FSImage (FSImage.java:run(795)) -
> Unable to save image for
> E:\HDFS-1623\hadoop-hdfs-project\hadoop-hdfs\build\test\data\dfs\name3
> java.io.IOException: saveLeases found path /test1/est/abc.txt but no matching
> entry in namespace.[/test1/est/abc.txt]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4336)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Saver.save(FSImageFormat.java:588)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:761)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:789)
> at java.lang.Thread.run(Unknown Source)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2827) Cannot save namespace after renaming a directory above a file with an open lease
[
https://issues.apache.org/jira/browse/HDFS-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197044#comment-13197044
]
Hudson commented on HDFS-2827:
--
Integrated in Hadoop-Common-trunk-Commit #1623 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1623/])
HDFS-2827. When the parent of a directory is the root, renaming the
directory results in leases updated incorrectly. Contributed by Uma Maheswara
Rao G
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238700
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
> Cannot save namespace after renaming a directory above a file with an open
> lease
>
>
> Key: HDFS-2827
> URL: https://issues.apache.org/jira/browse/HDFS-2827
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.24.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2827-test.patch, HDFS-2827.patch
>
>
> When i execute the following operations and wait for checkpoint to complete.
> fs.mkdirs(new Path("/test1"));
> FSDataOutputStream create = fs.create(new Path("/test/abc.txt")); //dont close
> fs.rename(new Path("/test/"), new Path("/test1/"));
> Check-pointing is failing with the following exception.
> 2012-01-23 15:03:14,204 ERROR namenode.FSImage (FSImage.java:run(795)) -
> Unable to save image for
> E:\HDFS-1623\hadoop-hdfs-project\hadoop-hdfs\build\test\data\dfs\name3
> java.io.IOException: saveLeases found path /test1/est/abc.txt but no matching
> entry in namespace.[/test1/est/abc.txt]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4336)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Saver.save(FSImageFormat.java:588)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:761)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:789)
> at java.lang.Thread.run(Unknown Source)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2742) HA: observed dataloss in replication stress test
[ https://issues.apache.org/jira/browse/HDFS-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2742: -- Attachment: hdfs-2742.txt Previous patch accidentally missed one of the earlier review passes by Eli (only enable incremental block tracking for ha NN). Re-incorporating it. > HA: observed dataloss in replication stress test > > > Key: HDFS-2742 > URL: https://issues.apache.org/jira/browse/HDFS-2742 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Blocker > Attachments: hdfs-2742.txt, hdfs-2742.txt, hdfs-2742.txt, > hdfs-2742.txt, hdfs-2742.txt, hdfs-2742.txt, hdfs-2742.txt, hdfs-2742.txt, > log-colorized.txt > > > The replication stress test case failed over the weekend since one of the > replicas went missing. Still diagnosing the issue, but it seems like the > chain of events was something like: > - a block report was generated on one of the nodes while the block was being > written - thus the block report listed the block as RBW > - when the standby replayed this queued message, it was replayed after the > file was marked complete. Thus it marked this replica as corrupt > - it asked the DN holding the corrupt replica to delete it. And, I think, > removed it from the block map at this time. > - That DN then did another block report before receiving the deletion. This > caused it to be re-added to the block map, since it was "FINALIZED" now. > - Replication was lowered on the file, and it counted the above replica as > non-corrupt, and asked for the other replicas to be deleted. > - All replicas were lost. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2827) Cannot save namespace after renaming a directory above a file with an open lease
[
https://issues.apache.org/jira/browse/HDFS-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197032#comment-13197032
]
Hudson commented on HDFS-2827:
--
Integrated in Hadoop-Hdfs-trunk-Commit #1695 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1695/])
HDFS-2827. When the parent of a directory is the root, renaming the
directory results in leases updated incorrectly. Contributed by Uma Maheswara
Rao G
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238700
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
> Cannot save namespace after renaming a directory above a file with an open
> lease
>
>
> Key: HDFS-2827
> URL: https://issues.apache.org/jira/browse/HDFS-2827
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.24.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2827-test.patch, HDFS-2827.patch
>
>
> When i execute the following operations and wait for checkpoint to complete.
> fs.mkdirs(new Path("/test1"));
> FSDataOutputStream create = fs.create(new Path("/test/abc.txt")); //dont close
> fs.rename(new Path("/test/"), new Path("/test1/"));
> Check-pointing is failing with the following exception.
> 2012-01-23 15:03:14,204 ERROR namenode.FSImage (FSImage.java:run(795)) -
> Unable to save image for
> E:\HDFS-1623\hadoop-hdfs-project\hadoop-hdfs\build\test\data\dfs\name3
> java.io.IOException: saveLeases found path /test1/est/abc.txt but no matching
> entry in namespace.[/test1/est/abc.txt]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4336)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Saver.save(FSImageFormat.java:588)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:761)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:789)
> at java.lang.Thread.run(Unknown Source)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2864) Remove redundant methods and a constant from FSDataset
[ https://issues.apache.org/jira/browse/HDFS-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2864: - Attachment: h2864_20120131.patch h2864_20120131.patch - removes the redundant methods and the constant. - adds FSDatasetInterface.contains(..) - synchronization is preserved. > Remove redundant methods and a constant from FSDataset > -- > > Key: HDFS-2864 > URL: https://issues.apache.org/jira/browse/HDFS-2864 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h2864_20120131.patch > > > - METADATA_VERSION is declared in both FSDataset and BlockMetadataHeader. > - In FSDataset, the methods findBlockFile(..), getBlockFile(..) and > getFile(..) are very similar. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2864) Remove redundant methods and a constant from FSDataset
[ https://issues.apache.org/jira/browse/HDFS-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2864: - Status: Patch Available (was: Open) > Remove redundant methods and a constant from FSDataset > -- > > Key: HDFS-2864 > URL: https://issues.apache.org/jira/browse/HDFS-2864 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h2864_20120131.patch > > > - METADATA_VERSION is declared in both FSDataset and BlockMetadataHeader. > - In FSDataset, the methods findBlockFile(..), getBlockFile(..) and > getFile(..) are very similar. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2815) Namenode is not coming out of safemode when we perform ( NN crash + restart ) . Also FSCK report shows blocks missed.
[ https://issues.apache.org/jira/browse/HDFS-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2815: -- Status: Patch Available (was: Open) > Namenode is not coming out of safemode when we perform ( NN crash + restart ) > . Also FSCK report shows blocks missed. > -- > > Key: HDFS-2815 > URL: https://issues.apache.org/jira/browse/HDFS-2815 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 1.0.0, 0.22.0, 0.24.0, 0.23.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Critical > Attachments: HDFS-2815.patch > > > When tested the HA(internal) with continuous switch with some 5mins gap, > found some *blocks missed* and namenode went into safemode after next switch. > >After the analysis, i found that this files already deleted by clients. > But i don't see any delete commands logs namenode log files. But namenode > added that blocks to invalidateSets and DNs deleted the blocks. >When restart of the namenode, it went into safemode and expecting some > more blocks to come out of safemode. >Here the reason could be that, file has been deleted in memory and added > into invalidates after this it is trying to sync the edits into editlog file. > By that time NN asked DNs to delete that blocks. Now namenode shuts down > before persisting to editlogs.( log behind) >Due to this reason, we may not get the INFO logs about delete, and when we > restart the Namenode (in my scenario it is again switch), Namenode expects > this deleted blocks also, as delete request is not persisted into editlog > before. >I reproduced this scenario with bedug points. *I feel, We should not add > the blocks to invalidates before persisting into Editlog*. > Note: for switch, we used kill -9 (force kill) > I am currently in 0.20.2 version. Same verified in 0.23 as well in normal > crash + restart scenario. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2864) Remove redundant methods and a constant from FSDataset
Remove redundant methods and a constant from FSDataset -- Key: HDFS-2864 URL: https://issues.apache.org/jira/browse/HDFS-2864 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE - METADATA_VERSION is declared in both FSDataset and BlockMetadataHeader. - In FSDataset, the methods findBlockFile(..), getBlockFile(..) and getFile(..) are very similar. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2827) Cannot save namespace after renaming a directory above a file with an open lease
[
https://issues.apache.org/jira/browse/HDFS-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197023#comment-13197023
]
Aaron T. Myers commented on HDFS-2827:
--
Nicholas, can you give me a day to review this before commit? I'd like to take
a look but don't have the time right this moment.
> Cannot save namespace after renaming a directory above a file with an open
> lease
>
>
> Key: HDFS-2827
> URL: https://issues.apache.org/jira/browse/HDFS-2827
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.24.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2827-test.patch, HDFS-2827.patch
>
>
> When i execute the following operations and wait for checkpoint to complete.
> fs.mkdirs(new Path("/test1"));
> FSDataOutputStream create = fs.create(new Path("/test/abc.txt")); //dont close
> fs.rename(new Path("/test/"), new Path("/test1/"));
> Check-pointing is failing with the following exception.
> 2012-01-23 15:03:14,204 ERROR namenode.FSImage (FSImage.java:run(795)) -
> Unable to save image for
> E:\HDFS-1623\hadoop-hdfs-project\hadoop-hdfs\build\test\data\dfs\name3
> java.io.IOException: saveLeases found path /test1/est/abc.txt but no matching
> entry in namespace.[/test1/est/abc.txt]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4336)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Saver.save(FSImageFormat.java:588)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:761)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:789)
> at java.lang.Thread.run(Unknown Source)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2827) Cannot save namespace after renaming a directory above a file with an open lease
[
https://issues.apache.org/jira/browse/HDFS-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-2827:
-
Hadoop Flags: Reviewed
+1 good catch.
> Cannot save namespace after renaming a directory above a file with an open
> lease
>
>
> Key: HDFS-2827
> URL: https://issues.apache.org/jira/browse/HDFS-2827
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.24.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2827-test.patch, HDFS-2827.patch
>
>
> When i execute the following operations and wait for checkpoint to complete.
> fs.mkdirs(new Path("/test1"));
> FSDataOutputStream create = fs.create(new Path("/test/abc.txt")); //dont close
> fs.rename(new Path("/test/"), new Path("/test1/"));
> Check-pointing is failing with the following exception.
> 2012-01-23 15:03:14,204 ERROR namenode.FSImage (FSImage.java:run(795)) -
> Unable to save image for
> E:\HDFS-1623\hadoop-hdfs-project\hadoop-hdfs\build\test\data\dfs\name3
> java.io.IOException: saveLeases found path /test1/est/abc.txt but no matching
> entry in namespace.[/test1/est/abc.txt]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4336)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Saver.save(FSImageFormat.java:588)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:761)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:789)
> at java.lang.Thread.run(Unknown Source)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2835) Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue
[ https://issues.apache.org/jira/browse/HDFS-2835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196956#comment-13196956 ] Robert Joseph Evans commented on HDFS-2835: --- I did not intend to have the other code in the patch. Findbugs gave me some extra warnings that went away after doing a clean and rerunning findbugs, but not before I started trying to fix those other warnings. Thanks for catching that. I like your patch it fixes the issue just as well as my patch. In either case this is just defensive programming as that enum is never actually serialized in the code. +1. > Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue > -- > > Key: HDFS-2835 > URL: https://issues.apache.org/jira/browse/HDFS-2835 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Attachments: HDFS-2835.txt, HDFS-2835.txt > > > https://builds.apache.org/job/PreCommit-HDFS-Build/1804//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html > shows a findbugs warning. It is unrelated to the patch being tested, and > has shown up on a few other JIRAS as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2691) HA: Tests and fixes for pipeline targets and replica recovery
[ https://issues.apache.org/jira/browse/HDFS-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196883#comment-13196883 ] Hudson commented on HDFS-2691: -- Integrated in Hadoop-Hdfs-HAbranch-build #64 (See [https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/64/]) HDFS-2691. Fixes for pipeline recovery in an HA cluster: report RBW replicas immediately upon pipeline creation. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237935 Files : * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-1623.txt * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/ReceivedDeletedBlockInfo.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/ReceivedDeletedBlockInfoWritable.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/AppendTestUtil.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestPipelinesFailover.java > HA: Tests and fixes for pipeline targets and replica recovery > - > > Key: HDFS-2691 > URL: https://issues.apache.org/jira/browse/HDFS-2691 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: HA branch (HDFS-1623) > > Attachments: hdfs-2691.txt, hdfs-2691.txt, hdfs-2691.txt, > hdfs-2691.txt > > > Currently there are some TODOs around pipeline/recovery code in the HA > branch. For example, commitBlockSynchronization only gets sent to the active > NN which may have failed over by that point. So, we need to write some tests > here and figure out what the correct behavior is. > Another related area is the treatment of targets in the pipeline. When a > pipeline is created, the active NN adds the "expected locations" to the > BlockInfoUnderConstruction, but the DN identifiers aren't logged with the > OP_ADD. So after a failover, the BlockInfoUnderConstruction will have no > targets and I imagine replica recovery would probably trigger some issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2853) HA: NN fails to start if the shared edits dir is marked required
[ https://issues.apache.org/jira/browse/HDFS-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196886#comment-13196886 ] Hudson commented on HDFS-2853: -- Integrated in Hadoop-Hdfs-HAbranch-build #64 (See [https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/64/]) HDFS-2853. HA: NN fails to start if the shared edits dir is marked required. Contributed by Aaron T. Myers. eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1238134 Files : * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-1623.txt * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourcePolicy.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeResourcePolicy.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java > HA: NN fails to start if the shared edits dir is marked required > > > Key: HDFS-2853 > URL: https://issues.apache.org/jira/browse/HDFS-2853 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Critical > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-2853-HDFS-1623.patch, HDFS-2853-HDFS-1623.patch > > > Currently it will fail because of a bug with checking the valid configuration > of required resources. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2841) HA: HAAdmin does not work if security is enabled
[
https://issues.apache.org/jira/browse/HDFS-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196887#comment-13196887
]
Hudson commented on HDFS-2841:
--
Integrated in Hadoop-Hdfs-HAbranch-build #64 (See
[https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/64/])
Amend HDFS-2841 to include new file which was omitted from original commit.
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1237971
Files :
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSHAAdmin.java
> HA: HAAdmin does not work if security is enabled
>
>
> Key: HDFS-2841
> URL: https://issues.apache.org/jira/browse/HDFS-2841
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
>Priority: Critical
> Fix For: HA branch (HDFS-1623)
>
> Attachments: HDFS-2841-HDFS-1623.patch
>
>
> Though HAServiceProtocol is indeed annotated with the {{@KerberosInfo}}
> annotation, it specifies the Common server principal, which is intended to be
> overridden at run-time with the value of the correct principal (e.g. as is
> done in MRAdmin and DFSAdmin.) We need to do something similar for HAAdmin.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
