[jira] [Commented] (HDFS-2941) Add an administrative command to download a copy of the fsimage from the NN
[ https://issues.apache.org/jira/browse/HDFS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238128#comment-13238128 ] Todd Lipcon commented on HDFS-2941: --- +1 pending jenkins results Add an administrative command to download a copy of the fsimage from the NN --- Key: HDFS-2941 URL: https://issues.apache.org/jira/browse/HDFS-2941 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch It would be nice to be able to download a copy of the fsimage from the NN, e.g. for backup purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2413) Add public APIs for safemode
[ https://issues.apache.org/jira/browse/HDFS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-2413: -- Attachment: HDFS-2413.patch Silly me (Applied changes onto the wrong class). Here's a new patch that cleans the additions up, and applies Nicholas' comments as well, properly this time. Add public APIs for safemode Key: HDFS-2413 URL: https://issues.apache.org/jira/browse/HDFS-2413 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Attachments: HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2413) Add public APIs for safemode
[ https://issues.apache.org/jira/browse/HDFS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238134#comment-13238134 ] Aaron T. Myers commented on HDFS-2413: -- Thanks, Harsh. The latest patch looks good to me. Nicholas, does it seem OK to you? Also, might it be reasonable to mark just isInSafeMode @InterfaceStability.Stable? Add public APIs for safemode Key: HDFS-2413 URL: https://issues.apache.org/jira/browse/HDFS-2413 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Attachments: HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2941) Add an administrative command to download a copy of the fsimage from the NN
[ https://issues.apache.org/jira/browse/HDFS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238139#comment-13238139 ] Hadoop QA commented on HDFS-2941: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519907/HDFS-2941.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.server.namenode.TestStartup org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs org.apache.hadoop.hdfs.TestGetBlocks org.apache.hadoop.hdfs.server.namenode.TestStorageRestore org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2096//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2096//console This message is automatically generated. Add an administrative command to download a copy of the fsimage from the NN --- Key: HDFS-2941 URL: https://issues.apache.org/jira/browse/HDFS-2941 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch It would be nice to be able to download a copy of the fsimage from the NN, e.g. for backup purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2941) Add an administrative command to download a copy of the fsimage from the NN
[ https://issues.apache.org/jira/browse/HDFS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2941: - Attachment: HDFS-2941.patch Turns out some of the tests end up calling TransferFsImage#downloadImageToStorage with localPaths set to null. The only difference between this patch and the last is an extra null check in TransferFsImage#downloadImageToStorage. This should address the test failures. Unless there are further comments, I'm going to commit this pending Jenkins based on Todd's earlier +1, since the difference between this patch and the last is trivial. Add an administrative command to download a copy of the fsimage from the NN --- Key: HDFS-2941 URL: https://issues.apache.org/jira/browse/HDFS-2941 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch It would be nice to be able to download a copy of the fsimage from the NN, e.g. for backup purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2185: -- Attachment: zkfc-design.pdf Attached a brief design doc that explains this from an overall perspective (including the HealthMonitor and ActiveStandbyElector work already committed) HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2941) Add an administrative command to download a copy of the fsimage from the NN
[ https://issues.apache.org/jira/browse/HDFS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238170#comment-13238170 ] Todd Lipcon commented on HDFS-2941: --- Seems reasonable, thanks. Add an administrative command to download a copy of the fsimage from the NN --- Key: HDFS-2941 URL: https://issues.apache.org/jira/browse/HDFS-2941 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch It would be nice to be able to download a copy of the fsimage from the NN, e.g. for backup purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2413) Add public APIs for safemode
[ https://issues.apache.org/jira/browse/HDFS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238185#comment-13238185 ] Hadoop QA commented on HDFS-2413: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519909/HDFS-2413.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestGetBlocks org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2097//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2097//console This message is automatically generated. Add public APIs for safemode Key: HDFS-2413 URL: https://issues.apache.org/jira/browse/HDFS-2413 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Attachments: HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2941) Add an administrative command to download a copy of the fsimage from the NN
[ https://issues.apache.org/jira/browse/HDFS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238197#comment-13238197 ] Hadoop QA commented on HDFS-2941: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519913/HDFS-2941.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestGetBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2098//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2098//console This message is automatically generated. Add an administrative command to download a copy of the fsimage from the NN --- Key: HDFS-2941 URL: https://issues.apache.org/jira/browse/HDFS-2941 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch It would be nice to be able to download a copy of the fsimage from the NN, e.g. for backup purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3152) Reading consistency for all readers
Reading consistency for all readers Key: HDFS-3152 URL: https://issues.apache.org/jira/browse/HDFS-3152 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.21.0, 0.20.2 Reporter: Denny Ye I met an exception when I would like to seek to latest size of file that another client was writing. Message is Cannot seek after EOF. I got the seek target from previous input stream and now I trying to obtains the file incremental. It means the target over than the file size limitation. In my opinion, the confirmed visible file length comes from the completed blocks(NameNode) plus replied size in last DataNode of pipeline for last block. Here are two cases: 1. How to obtains the confirmed visible file length to all readers. 2. For each reader, how can we pick out the best DN for concrete block. Actually, existing code mix up those two parts. NameNode sorted block locations due to local reading(HBase or local MapReduce, random DataNode for outer reader). DFSClient obtains the first DataNode of last block. Pay attention to this point! Client may obtains the 'dirty' file length from frist DN of last block that NameNode returned. And client always uses the frist DN for each block to read file content. Should we split two cases? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2941) Add an administrative command to download a copy of the fsimage from the NN
[ https://issues.apache.org/jira/browse/HDFS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2941: - Resolution: Fixed Fix Version/s: 0.23.3 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-0.23. Thanks a lot for the reviews, Todd. Add an administrative command to download a copy of the fsimage from the NN --- Key: HDFS-2941 URL: https://issues.apache.org/jira/browse/HDFS-2941 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.23.3 Attachments: HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch It would be nice to be able to download a copy of the fsimage from the NN, e.g. for backup purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2941) Add an administrative command to download a copy of the fsimage from the NN
[ https://issues.apache.org/jira/browse/HDFS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238580#comment-13238580 ] Hudson commented on HDFS-2941: -- Integrated in Hadoop-Hdfs-0.23-Commit #718 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/718/]) HDFS-2941. Add an administrative command to download a copy of the fsimage from the NN. Contributed by Aaron T. Myers. (Revision 1305450) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305450 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GetImageServlet.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFetchImage.java Add an administrative command to download a copy of the fsimage from the NN --- Key: HDFS-2941 URL: https://issues.apache.org/jira/browse/HDFS-2941 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.23.3 Attachments: HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch It would be nice to be able to download a copy of the fsimage from the NN, e.g. for backup purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2941) Add an administrative command to download a copy of the fsimage from the NN
[ https://issues.apache.org/jira/browse/HDFS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238581#comment-13238581 ] Hudson commented on HDFS-2941: -- Integrated in Hadoop-Hdfs-trunk-Commit #2004 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2004/]) Move CHANGES.txt entry for HDFS-2941 to the right version. (Revision 1305453) HDFS-2941. Add an administrative command to download a copy of the fsimage from the NN. Contributed by Aaron T. Myers. (Revision 1305447) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305453 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305447 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GetImageServlet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFetchImage.java Add an administrative command to download a copy of the fsimage from the NN --- Key: HDFS-2941 URL: https://issues.apache.org/jira/browse/HDFS-2941 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.23.3 Attachments: HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch It would be nice to be able to download a copy of the fsimage from the NN, e.g. for backup purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2941) Add an administrative command to download a copy of the fsimage from the NN
[ https://issues.apache.org/jira/browse/HDFS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238582#comment-13238582 ] Hudson commented on HDFS-2941: -- Integrated in Hadoop-Common-0.23-Commit #728 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/728/]) HDFS-2941. Add an administrative command to download a copy of the fsimage from the NN. Contributed by Aaron T. Myers. (Revision 1305450) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305450 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GetImageServlet.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFetchImage.java Add an administrative command to download a copy of the fsimage from the NN --- Key: HDFS-2941 URL: https://issues.apache.org/jira/browse/HDFS-2941 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.23.3 Attachments: HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch It would be nice to be able to download a copy of the fsimage from the NN, e.g. for backup purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2413) Add public APIs for safemode
[ https://issues.apache.org/jira/browse/HDFS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238593#comment-13238593 ] Tsz Wo (Nicholas), SZE commented on HDFS-2413: -- +1 Thanks Harsh, the patch looks good. Also, might it be reasonable to mark just isInSafeMode @InterfaceStability.Stable? Since this is a new API, we are not sure if want to change it. Let's leave it as-is for the moment. Add public APIs for safemode Key: HDFS-2413 URL: https://issues.apache.org/jira/browse/HDFS-2413 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Attachments: HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2941) Add an administrative command to download a copy of the fsimage from the NN
[ https://issues.apache.org/jira/browse/HDFS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238600#comment-13238600 ] Hudson commented on HDFS-2941: -- Integrated in Hadoop-Common-trunk-Commit #1930 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1930/]) Move CHANGES.txt entry for HDFS-2941 to the right version. (Revision 1305453) HDFS-2941. Add an administrative command to download a copy of the fsimage from the NN. Contributed by Aaron T. Myers. (Revision 1305447) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305453 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305447 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GetImageServlet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFetchImage.java Add an administrative command to download a copy of the fsimage from the NN --- Key: HDFS-2941 URL: https://issues.apache.org/jira/browse/HDFS-2941 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.23.3 Attachments: HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch It would be nice to be able to download a copy of the fsimage from the NN, e.g. for backup purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3125) Add a service that enables JournalDaemon
[ https://issues.apache.org/jira/browse/HDFS-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238611#comment-13238611 ] Suresh Srinivas commented on HDFS-3125: --- bq. The new patch needs to check state in journal(..) and startLogSegment(..), and verify(registration) in startLogSegment(..). Otherwise, it looks good. Currently theses states are for listener side only. The method calls above are received over RPC call. Given that RPC server could be controlled from outside, the expectation is that either Service stop has stopped RPC server or external application has stopped the server. The missing functionality currently is - Service should unregister during stop. Currently an errorReport of fatal is used as unregister. I was planning to add unregister method to the protocol. If that is not needed, I can call errorReport in service#stop() method. Add a service that enables JournalDaemon Key: HDFS-3125 URL: https://issues.apache.org/jira/browse/HDFS-3125 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-3125.patch, HDFS-3125.patch, HDFS-3125.patch In this subtask, I plan to add JournalService. It will provide the following functionality: # Starts RPC server with JournalProtocolService or uses the RPC server provided and add JournalProtocol service. # Registers with the namenode. # Receives JournalProtocol related requests and hands it to over to a listener. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238629#comment-13238629 ] Bikas Saha commented on HDFS-2185: -- Thanks! I will look at the doc. ramblingI agree that the best documentation is the code. But that is for the developers. Putting up a design document helps people not familiar with the code or who dont have the time to extract the design from the code, to contribute potentially valuable inputs. This is specially true for a new piece of code like this one, where there is not much previous background to rely on. Once this is committed, then the code would be the best documentation. Updating docs would be good but is usually not practical. I have learned that the more I do to help my reviewers, the better reviews I get. E.g. if I were doing it, in this case, I would also include a diagram that briefly sketches the code/object organization/relationships with tips on where to start so that my review process is made easy./rambling HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3050) refactor OEV to share more code with the NameNode
[ https://issues.apache.org/jira/browse/HDFS-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3050: --- Attachment: HDFS-3050.012.patch * fix findbugs warnings refactor OEV to share more code with the NameNode - Key: HDFS-3050 URL: https://issues.apache.org/jira/browse/HDFS-3050 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-3050.006.patch, HDFS-3050.007.patch, HDFS-3050.008.patch, HDFS-3050.009.patch, HDFS-3050.010.patch, HDFS-3050.011.patch, HDFS-3050.012.patch Current, OEV (the offline edits viewer) re-implements all of the opcode parsing logic found in the NameNode. This duplicated code creates a maintenance burden for us. OEV should be refactored to simply use the normal EditLog parsing code, rather than rolling its own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2941) Add an administrative command to download a copy of the fsimage from the NN
[ https://issues.apache.org/jira/browse/HDFS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238642#comment-13238642 ] Hudson commented on HDFS-2941: -- Integrated in Hadoop-Mapreduce-0.23-Commit #737 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/737/]) HDFS-2941. Add an administrative command to download a copy of the fsimage from the NN. Contributed by Aaron T. Myers. (Revision 1305450) Result = ABORTED atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305450 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GetImageServlet.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFetchImage.java Add an administrative command to download a copy of the fsimage from the NN --- Key: HDFS-2941 URL: https://issues.apache.org/jira/browse/HDFS-2941 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.23.3 Attachments: HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch It would be nice to be able to download a copy of the fsimage from the NN, e.g. for backup purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2941) Add an administrative command to download a copy of the fsimage from the NN
[ https://issues.apache.org/jira/browse/HDFS-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238643#comment-13238643 ] Hudson commented on HDFS-2941: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1940 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1940/]) Move CHANGES.txt entry for HDFS-2941 to the right version. (Revision 1305453) HDFS-2941. Add an administrative command to download a copy of the fsimage from the NN. Contributed by Aaron T. Myers. (Revision 1305447) Result = ABORTED atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305453 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305447 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GetImageServlet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFetchImage.java Add an administrative command to download a copy of the fsimage from the NN --- Key: HDFS-2941 URL: https://issues.apache.org/jira/browse/HDFS-2941 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client, name-node Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.23.3 Attachments: HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch, HDFS-2941.patch It would be nice to be able to download a copy of the fsimage from the NN, e.g. for backup purposes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2413) Add public APIs for safemode
[ https://issues.apache.org/jira/browse/HDFS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238649#comment-13238649 ] Aaron T. Myers commented on HDFS-2413: -- bq. Since this is a new API, we are not sure if want to change it. Let's leave it as-is for the moment. Works for me. Thanks for thinking about it. Nicholas, would you like to commit this patch or shall I? We should also probably update the target version to 0.23.3 instead of 0.23.2. Add public APIs for safemode Key: HDFS-2413 URL: https://issues.apache.org/jira/browse/HDFS-2413 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Attachments: HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2656) Implement a pure c client based on webhdfs
[ https://issues.apache.org/jira/browse/HDFS-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238651#comment-13238651 ] Zhanwei.Wang commented on HDFS-2656: Hi everyone, The code of this jira proposed is almost finished and will available soon. It is more complicated than what I thought and cost me more time. The following is the current status of the code. Status update: 1, finished: most functions in hdfs.h of libhdfs are implemented. 2, on-going: function test. unit test. document 3, todo: kerberos support. http proxy support. some performance improvement Implement a pure c client based on webhdfs -- Key: HDFS-2656 URL: https://issues.apache.org/jira/browse/HDFS-2656 Project: Hadoop HDFS Issue Type: Improvement Reporter: Zhanwei.Wang Currently, the implementation of libhdfs is based on JNI. The overhead of JVM seems a little big, and libhdfs can also not be used in the environment without hdfs. It seems a good idea to implement a pure c client by wrapping webhdfs. It also can be used to access different version of hdfs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3004) Implement Recovery Mode
[
https://issues.apache.org/jira/browse/HDFS-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238672#comment-13238672
]
Colin Patrick McCabe commented on HDFS-3004:
It looks like the findbugs warning relates to NameNode::state. As far as I can
see, I haven't changed how the HAState stuff works at all, so I don't think
this has anything to do with the patch.
Test failues:
testCheckpointNode: a port-in-use problem
testBackupNodeTailsEdits: a port-in-use problem
testBackupNode: a port-in-use problem
TestHDFSCLI: a lot of lines similar to this:
{code}
Expected output: [setSpaceQuota: java.io.FileNotFoundException: Directory
does not exist: /test1]
Actual output: [setSpaceQuota: Directory does not exist: /test1
{code}
But when I run the test, it passes. Also, when I run the test manually, I do
see the expected exception message:
{code}
cmccabe@keter:/opt/hadoop/home4 ./bin/hadoop dfsadmin -setSpaceQuota 1g /test1
setSpaceQuota: java.io.FileNotFoundException: Directory does not exist: /test1
{code}
TestGetBlocks.testGetBlocks: test works fine when I run it locally. Flaky test?
Implement Recovery Mode
---
Key: HDFS-3004
URL: https://issues.apache.org/jira/browse/HDFS-3004
Project: Hadoop HDFS
Issue Type: New Feature
Components: tools
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-3004.010.patch, HDFS-3004.011.patch,
HDFS-3004.012.patch, HDFS-3004.013.patch, HDFS-3004.015.patch,
HDFS-3004.016.patch, HDFS-3004.017.patch, HDFS-3004.018.patch,
HDFS-3004.019.patch, HDFS-3004.020.patch, HDFS-3004.022.patch,
HDFS-3004.023.patch, HDFS-3004.024.patch, HDFS-3004.026.patch,
HDFS-3004.027.patch, HDFS-3004.029.patch, HDFS-3004.030.patch,
HDFS-3004.031.patch, HDFS-3004.032.patch,
HDFS-3004__namenode_recovery_tool.txt
When the NameNode metadata is corrupt for some reason, we want to be able to
fix it. Obviously, we would prefer never to get in this case. In a perfect
world, we never would. However, bad data on disk can happen from time to
time, because of hardware errors or misconfigurations. In the past we have
had to correct it manually, which is time-consuming and which can result in
downtime.
Recovery mode is initialized by the system administrator. When the NameNode
starts up in Recovery Mode, it will try to load the FSImage file, apply all
the edits from the edits log, and then write out a new image. Then it will
shut down.
Unlike in the normal startup process, the recovery mode startup process will
be interactive. When the NameNode finds something that is inconsistent, it
will prompt the operator as to what it should do. The operator can also
choose to take the first option for all prompts by starting up with the '-f'
flag, or typing 'a' at one of the prompts.
I have reused as much code as possible from the NameNode in this tool.
Hopefully, the effort that was spent developing this will also make the
NameNode editLog and image processing even more robust than it already is.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3050) refactor OEV to share more code with the NameNode
[ https://issues.apache.org/jira/browse/HDFS-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238712#comment-13238712 ] Hadoop QA commented on HDFS-3050: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519986/HDFS-3050.012.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. -1 javac. The applied patch generated 20 javac compiler warnings (more than the trunk's current 14 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestGetBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2099//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2099//console This message is automatically generated. refactor OEV to share more code with the NameNode - Key: HDFS-3050 URL: https://issues.apache.org/jira/browse/HDFS-3050 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-3050.006.patch, HDFS-3050.007.patch, HDFS-3050.008.patch, HDFS-3050.009.patch, HDFS-3050.010.patch, HDFS-3050.011.patch, HDFS-3050.012.patch Current, OEV (the offline edits viewer) re-implements all of the opcode parsing logic found in the NameNode. This duplicated code creates a maintenance burden for us. OEV should be refactored to simply use the normal EditLog parsing code, rather than rolling its own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3089) Move FSDatasetInterface and other related classes/interfaces to a package
[
https://issues.apache.org/jira/browse/HDFS-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238723#comment-13238723
]
Uma Maheswara Rao G commented on HDFS-3089:
---
Hi Nicholas,
I have reviewed h3089_20120321b.patch.
Patch mostly looks good. A few minor comments.
1) looks you have removed dfs.datanode.block.volume.choice.policy config item
from hdfs-default.xml
may need to update with correct package name right? if there is a reason for
removal or some other discussion about it, could you please explain?. If we
really want to remove the default config item, this JIRA may not be the correct
place right?
2) In my opinion, we may need to maintain the consistency of the class Names
{code}
public class FSDataset implements FsDatasetSpiFSDataset.FSVolume {
{code}
Many places we are reffering FileSystem as FS ex: FSnamesystem,
FSDataset.
Now we named the SPI with small 's'.
Thanks
Uma
Move FSDatasetInterface and other related classes/interfaces to a package
-
Key: HDFS-3089
URL: https://issues.apache.org/jira/browse/HDFS-3089
Project: Hadoop HDFS
Issue Type: Sub-task
Components: data-node
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Attachments: h3089_20120316_svn_mv.patch, h3089_20120319.patch,
h3089_20120319_svn_mv.patch, h3089_20120321.patch, h3089_20120321b.patch,
h3089_20120321b_svn_mv.patch, svn_mv.sh
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3153) For HA, a logical name is visible in URIs - add an explicit logical name
For HA, a logical name is visible in URIs - add an explicit logical name Key: HDFS-3153 URL: https://issues.apache.org/jira/browse/HDFS-3153 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3089) Move FSDatasetInterface and other related classes/interfaces to a package
[ https://issues.apache.org/jira/browse/HDFS-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238762#comment-13238762 ] Tsz Wo (Nicholas), SZE commented on HDFS-3089: -- Hi Uma, thanks for the review. For 1), I removed it since VolumeChoosingPolicy is annotated as private like BlockPlacementPolicy. So it should not be in hdfs-default.xml, which should contain only public APIs. Advanced users could add this property if they want. For 2), we actually do not yet have a consistent naming convention. For examples, we have FsPermission, FsStatus, FsShell, FsServerDefaults, o.a.h.f.FsConstants, etc. but also FSDataInputStream, FSDataOutputStream, FSNamesystem, FSImage, etc. We had [a discussion|https://issues.apache.org/jira/browse/HDFS-245?focusedCommentId=12637296page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12637296] and I filed HADOOP-4357 some time back. Most public APIs use FsXxx. I think that FsXxx is preferred over FSXxx. So new classes should be named FsXxx instead. Move FSDatasetInterface and other related classes/interfaces to a package - Key: HDFS-3089 URL: https://issues.apache.org/jira/browse/HDFS-3089 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h3089_20120316_svn_mv.patch, h3089_20120319.patch, h3089_20120319_svn_mv.patch, h3089_20120321.patch, h3089_20120321b.patch, h3089_20120321b_svn_mv.patch, svn_mv.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3153) For HA, a logical name is visible in URIs - add an explicit logical name
[ https://issues.apache.org/jira/browse/HDFS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-3153: --- Issue Type: Sub-task (was: Improvement) Parent: HDFS-1623 For HA, a logical name is visible in URIs - add an explicit logical name Key: HDFS-3153 URL: https://issues.apache.org/jira/browse/HDFS-3153 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Sanjay Radia -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3089) Move FSDatasetInterface and other related classes/interfaces to a package
[ https://issues.apache.org/jira/browse/HDFS-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238773#comment-13238773 ] Uma Maheswara Rao G commented on HDFS-3089: --- Thanks for the explanation. And i have seen the discussion in HDFS-245. Make sense to fix along with HADOOP-4357. I am +1 on the patch. Move FSDatasetInterface and other related classes/interfaces to a package - Key: HDFS-3089 URL: https://issues.apache.org/jira/browse/HDFS-3089 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h3089_20120316_svn_mv.patch, h3089_20120319.patch, h3089_20120319_svn_mv.patch, h3089_20120321.patch, h3089_20120321b.patch, h3089_20120321b_svn_mv.patch, svn_mv.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3050) refactor OEV to share more code with the NameNode
[ https://issues.apache.org/jira/browse/HDFS-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238788#comment-13238788 ] Eli Collins commented on HDFS-3050: --- Another patch for the TestOfflineEditsViewer failure? refactor OEV to share more code with the NameNode - Key: HDFS-3050 URL: https://issues.apache.org/jira/browse/HDFS-3050 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-3050.006.patch, HDFS-3050.007.patch, HDFS-3050.008.patch, HDFS-3050.009.patch, HDFS-3050.010.patch, HDFS-3050.011.patch, HDFS-3050.012.patch Current, OEV (the offline edits viewer) re-implements all of the opcode parsing logic found in the NameNode. This duplicated code creates a maintenance burden for us. OEV should be refactored to simply use the normal EditLog parsing code, rather than rolling its own. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3154) Add a notion of immutable/mutable files
Add a notion of immutable/mutable files --- Key: HDFS-3154 URL: https://issues.apache.org/jira/browse/HDFS-3154 Project: Hadoop HDFS Issue Type: New Feature Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE The notion of immutable file is useful since it lets the system and tools optimize certain things as discussed in [this email thread|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201203.mbox/%3CCAPn_vTuZomPmBTypP8_1xTr49Sj0fy7Mjhik4DbcAA+BLH53=g...@mail.gmail.com%3E]. Also, many applications require only immutable files. Here is a proposal: - Immutable files means that the file content is immutable. Operations such as append and truncate that change the file content are not allowed to act on immutable files. However, the meta data such as replication and permission of an immutable file can be updated. Immutable files can also be deleted or renamed. - Users have to pass immutable/mutable as a flag in file creation. This is an unmodifiable property of the created file. - If users want to change the data in an immutable file, the file could be copied to another file which is created as mutable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3153) For HA, a logical name is visible in URIs - add an explicit logical name
[ https://issues.apache.org/jira/browse/HDFS-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238803#comment-13238803 ] Aaron T. Myers commented on HDFS-3153: -- Hi Sanjay, given that HDFS-1623 is closed and the HA work has been merged to trunk, I think we should move this to be a top-level JIRA. Do you agree? For HA, a logical name is visible in URIs - add an explicit logical name Key: HDFS-3153 URL: https://issues.apache.org/jira/browse/HDFS-3153 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Sanjay Radia -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3004) Implement Recovery Mode
[ https://issues.apache.org/jira/browse/HDFS-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238823#comment-13238823 ] Aaron T. Myers commented on HDFS-3004: -- TestHDFSCLI and TestGetBlocks are known to be failing on trunk: see HDFS-3142 and HDFS-3143. If TestGetBlocks is passing for you, it's probably because you have an old snapshot of the source tree. Implement Recovery Mode --- Key: HDFS-3004 URL: https://issues.apache.org/jira/browse/HDFS-3004 Project: Hadoop HDFS Issue Type: New Feature Components: tools Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-3004.010.patch, HDFS-3004.011.patch, HDFS-3004.012.patch, HDFS-3004.013.patch, HDFS-3004.015.patch, HDFS-3004.016.patch, HDFS-3004.017.patch, HDFS-3004.018.patch, HDFS-3004.019.patch, HDFS-3004.020.patch, HDFS-3004.022.patch, HDFS-3004.023.patch, HDFS-3004.024.patch, HDFS-3004.026.patch, HDFS-3004.027.patch, HDFS-3004.029.patch, HDFS-3004.030.patch, HDFS-3004.031.patch, HDFS-3004.032.patch, HDFS-3004__namenode_recovery_tool.txt When the NameNode metadata is corrupt for some reason, we want to be able to fix it. Obviously, we would prefer never to get in this case. In a perfect world, we never would. However, bad data on disk can happen from time to time, because of hardware errors or misconfigurations. In the past we have had to correct it manually, which is time-consuming and which can result in downtime. Recovery mode is initialized by the system administrator. When the NameNode starts up in Recovery Mode, it will try to load the FSImage file, apply all the edits from the edits log, and then write out a new image. Then it will shut down. Unlike in the normal startup process, the recovery mode startup process will be interactive. When the NameNode finds something that is inconsistent, it will prompt the operator as to what it should do. The operator can also choose to take the first option for all prompts by starting up with the '-f' flag, or typing 'a' at one of the prompts. I have reused as much code as possible from the NameNode in this tool. Hopefully, the effort that was spent developing this will also make the NameNode editLog and image processing even more robust than it already is. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3004) Implement Recovery Mode
[ https://issues.apache.org/jira/browse/HDFS-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3004: --- Attachment: HDFS-3004.033.patch * rebase on trunk * fix some resource leaks in TestNameNodeRecovery * some whitespace and logging cleanups Implement Recovery Mode --- Key: HDFS-3004 URL: https://issues.apache.org/jira/browse/HDFS-3004 Project: Hadoop HDFS Issue Type: New Feature Components: tools Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-3004.010.patch, HDFS-3004.011.patch, HDFS-3004.012.patch, HDFS-3004.013.patch, HDFS-3004.015.patch, HDFS-3004.016.patch, HDFS-3004.017.patch, HDFS-3004.018.patch, HDFS-3004.019.patch, HDFS-3004.020.patch, HDFS-3004.022.patch, HDFS-3004.023.patch, HDFS-3004.024.patch, HDFS-3004.026.patch, HDFS-3004.027.patch, HDFS-3004.029.patch, HDFS-3004.030.patch, HDFS-3004.031.patch, HDFS-3004.032.patch, HDFS-3004.033.patch, HDFS-3004__namenode_recovery_tool.txt When the NameNode metadata is corrupt for some reason, we want to be able to fix it. Obviously, we would prefer never to get in this case. In a perfect world, we never would. However, bad data on disk can happen from time to time, because of hardware errors or misconfigurations. In the past we have had to correct it manually, which is time-consuming and which can result in downtime. Recovery mode is initialized by the system administrator. When the NameNode starts up in Recovery Mode, it will try to load the FSImage file, apply all the edits from the edits log, and then write out a new image. Then it will shut down. Unlike in the normal startup process, the recovery mode startup process will be interactive. When the NameNode finds something that is inconsistent, it will prompt the operator as to what it should do. The operator can also choose to take the first option for all prompts by starting up with the '-f' flag, or typing 'a' at one of the prompts. I have reused as much code as possible from the NameNode in this tool. Hopefully, the effort that was spent developing this will also make the NameNode editLog and image processing even more robust than it already is. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3154) Add a notion of immutable/mutable files
[ https://issues.apache.org/jira/browse/HDFS-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238841#comment-13238841 ] Scott Carey commented on HDFS-3154: --- You should be able to make a copy of an immutable file that takes up no extra space, but can be appended to or truncated. The immutable file and the copy would share immutable blocks. Add a notion of immutable/mutable files --- Key: HDFS-3154 URL: https://issues.apache.org/jira/browse/HDFS-3154 Project: Hadoop HDFS Issue Type: New Feature Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE The notion of immutable file is useful since it lets the system and tools optimize certain things as discussed in [this email thread|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201203.mbox/%3CCAPn_vTuZomPmBTypP8_1xTr49Sj0fy7Mjhik4DbcAA+BLH53=g...@mail.gmail.com%3E]. Also, many applications require only immutable files. Here is a proposal: - Immutable files means that the file content is immutable. Operations such as append and truncate that change the file content are not allowed to act on immutable files. However, the meta data such as replication and permission of an immutable file can be updated. Immutable files can also be deleted or renamed. - Users have to pass immutable/mutable as a flag in file creation. This is an unmodifiable property of the created file. - If users want to change the data in an immutable file, the file could be copied to another file which is created as mutable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3089) Move FSDatasetInterface and other related classes/interfaces to a package
[ https://issues.apache.org/jira/browse/HDFS-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3089: - Resolution: Fixed Fix Version/s: 0.23.3 0.24.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I have committed this. Move FSDatasetInterface and other related classes/interfaces to a package - Key: HDFS-3089 URL: https://issues.apache.org/jira/browse/HDFS-3089 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.24.0, 0.23.3 Attachments: h3089_20120316_svn_mv.patch, h3089_20120319.patch, h3089_20120319_svn_mv.patch, h3089_20120321.patch, h3089_20120321b.patch, h3089_20120321b_svn_mv.patch, svn_mv.sh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3125) Add a service that enables JournalDaemon
[ https://issues.apache.org/jira/browse/HDFS-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-3125: -- Attachment: HDFS-3125.patch Updated patch with verifyRequest added to startLogSegment method Add a service that enables JournalDaemon Key: HDFS-3125 URL: https://issues.apache.org/jira/browse/HDFS-3125 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-3125.patch, HDFS-3125.patch, HDFS-3125.patch, HDFS-3125.patch In this subtask, I plan to add JournalService. It will provide the following functionality: # Starts RPC server with JournalProtocolService or uses the RPC server provided and add JournalProtocol service. # Registers with the namenode. # Receives JournalProtocol related requests and hands it to over to a listener. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3154) Add a notion of immutable/mutable files
[ https://issues.apache.org/jira/browse/HDFS-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238868#comment-13238868 ] Colin Patrick McCabe commented on HDFS-3154: I am aware of at least two filesystems (in the loose sense of the word) that have immutable files. One is Venti from Plan9, and the other is git, by Linus Torvalds. Both of them are significantly simpler because of their invariant that files cannot change. However, both of them are append-only, meaning that files can never be deleted. This seems unsuitable for the HDFS use case, and in fact, I see no reason to believe that having some, but not all, files be immutable would provide any benefit. Can you list the concrete benefits you think would come from having some, but not all, files be immutable? The linked email thread does not list anything specific. Add a notion of immutable/mutable files --- Key: HDFS-3154 URL: https://issues.apache.org/jira/browse/HDFS-3154 Project: Hadoop HDFS Issue Type: New Feature Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE The notion of immutable file is useful since it lets the system and tools optimize certain things as discussed in [this email thread|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201203.mbox/%3CCAPn_vTuZomPmBTypP8_1xTr49Sj0fy7Mjhik4DbcAA+BLH53=g...@mail.gmail.com%3E]. Also, many applications require only immutable files. Here is a proposal: - Immutable files means that the file content is immutable. Operations such as append and truncate that change the file content are not allowed to act on immutable files. However, the meta data such as replication and permission of an immutable file can be updated. Immutable files can also be deleted or renamed. - Users have to pass immutable/mutable as a flag in file creation. This is an unmodifiable property of the created file. - If users want to change the data in an immutable file, the file could be copied to another file which is created as mutable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3131) Improve TestStorageRestore
[ https://issues.apache.org/jira/browse/HDFS-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-3131: - Attachment: HDFS-3131.patch Improve TestStorageRestore -- Key: HDFS-3131 URL: https://issues.apache.org/jira/browse/HDFS-3131 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.24.0, 1.1.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Brandon Li Priority: Minor Labels: newbie Attachments: HDFS-3131.branch-1.patch, HDFS-3131.patch, HDFS-3131.patch Aaron has the following comments on TestStorageRestore in HDFS-3127. # removeStorageAccess, restoreAccess, and numStorageDirs can all be made private # numStorageDirs can be made static # Rather than do set(Readable/Executable/Writable), use FileUtil.chmod(...). # Please put the contents of the test in a try/finally, with the calls to shutdown the cluster and the 2NN in the finally block. # Some lines are over 80 chars. # No need for the numDatanodes variable - it's only used in one place. # Instead of xwr use rwx, which I think is a more common way of describing permissions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3131) Improve TestStorageRestore
[ https://issues.apache.org/jira/browse/HDFS-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238880#comment-13238880 ] Brandon Li commented on HDFS-3131: -- The test failures and warnings are not caused by this patch. The warnings were fixed in HDFS-3132. Aaron, could you please review the patch? Thanks, Brandon Improve TestStorageRestore -- Key: HDFS-3131 URL: https://issues.apache.org/jira/browse/HDFS-3131 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.24.0, 1.1.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Brandon Li Priority: Minor Labels: newbie Attachments: HDFS-3131.branch-1.patch, HDFS-3131.patch, HDFS-3131.patch Aaron has the following comments on TestStorageRestore in HDFS-3127. # removeStorageAccess, restoreAccess, and numStorageDirs can all be made private # numStorageDirs can be made static # Rather than do set(Readable/Executable/Writable), use FileUtil.chmod(...). # Please put the contents of the test in a try/finally, with the calls to shutdown the cluster and the 2NN in the finally block. # Some lines are over 80 chars. # No need for the numDatanodes variable - it's only used in one place. # Instead of xwr use rwx, which I think is a more common way of describing permissions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3129) NetworkTopology: add test that getLeaf should check for invalid topologies
[ https://issues.apache.org/jira/browse/HDFS-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3129: -- Resolution: Fixed Fix Version/s: 0.23.3 1.1.0 Target Version/s: (was: 0.23.3, 1.1.0) Status: Resolved (was: Patch Available) Test failures are unrelated. I've committed this and merged to branch-23 and branch-1. Thanks Colin! NetworkTopology: add test that getLeaf should check for invalid topologies -- Key: HDFS-3129 URL: https://issues.apache.org/jira/browse/HDFS-3129 Project: Hadoop HDFS Issue Type: Test Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Fix For: 1.1.0, 0.23.3 Attachments: HDFS-3129-b1.001.patch, HDFS-3129.001.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2413) Add public APIs for safemode
[ https://issues.apache.org/jira/browse/HDFS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2413: - Resolution: Fixed Fix Version/s: 0.23.3 0.24.0 Target Version/s: 0.24.0, 0.23.2 (was: 0.23.2, 0.24.0) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I have committed this. Thanks, Harsh! Add public APIs for safemode Key: HDFS-2413 URL: https://issues.apache.org/jira/browse/HDFS-2413 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Fix For: 0.24.0, 0.23.3 Attachments: HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3004) Implement Recovery Mode
[ https://issues.apache.org/jira/browse/HDFS-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238918#comment-13238918 ] Hadoop QA commented on HDFS-3004: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520009/HDFS-3004.033.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 21 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestGetBlocks org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2100//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2100//console This message is automatically generated. Implement Recovery Mode --- Key: HDFS-3004 URL: https://issues.apache.org/jira/browse/HDFS-3004 Project: Hadoop HDFS Issue Type: New Feature Components: tools Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-3004.010.patch, HDFS-3004.011.patch, HDFS-3004.012.patch, HDFS-3004.013.patch, HDFS-3004.015.patch, HDFS-3004.016.patch, HDFS-3004.017.patch, HDFS-3004.018.patch, HDFS-3004.019.patch, HDFS-3004.020.patch, HDFS-3004.022.patch, HDFS-3004.023.patch, HDFS-3004.024.patch, HDFS-3004.026.patch, HDFS-3004.027.patch, HDFS-3004.029.patch, HDFS-3004.030.patch, HDFS-3004.031.patch, HDFS-3004.032.patch, HDFS-3004.033.patch, HDFS-3004__namenode_recovery_tool.txt When the NameNode metadata is corrupt for some reason, we want to be able to fix it. Obviously, we would prefer never to get in this case. In a perfect world, we never would. However, bad data on disk can happen from time to time, because of hardware errors or misconfigurations. In the past we have had to correct it manually, which is time-consuming and which can result in downtime. Recovery mode is initialized by the system administrator. When the NameNode starts up in Recovery Mode, it will try to load the FSImage file, apply all the edits from the edits log, and then write out a new image. Then it will shut down. Unlike in the normal startup process, the recovery mode startup process will be interactive. When the NameNode finds something that is inconsistent, it will prompt the operator as to what it should do. The operator can also choose to take the first option for all prompts by starting up with the '-f' flag, or typing 'a' at one of the prompts. I have reused as much code as possible from the NameNode in this tool. Hopefully, the effort that was spent developing this will also make the NameNode editLog and image processing even more robust than it already is. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238922#comment-13238922 ] Aaron T. Myers commented on HDFS-2185: -- Patch looks pretty good, Todd. A few small comments: # Is it intentional that you call super.setConf(...) in DFSZKFailoverController#setConf with the passed-conf as-is, instead of the slightly mutated localNNConf? If it is intentional, then please move the super.setConf(...) call higher in the method to make that clear. # Is it necessary for DFSZKFailoverController#getLocalTarget to return a new NNHAServiceTarget object each time it is called? If so, please add a comment explaining why. If not, then let's just store a single instance of NNHAServiceTarget instead of the separate conf/nsId/nnId instance variables. # I think you might end up with a race condition if you don't add a waitForHAState(1, HAServiceState.STANDBY) in TestDFSZKFailoverController#setup after starting the second ZKFCThread. # A little too much indentation at the end of TestDFSZKFailoverController#setup. # After the call to cluster.restartNameNode(0), I think we should assert that the second NN is still active. As-written, the test could succeed if the system failed back to the first NN upon restarting it. This assertion is sort of more a test of the common side of this patch, though, so feel free to punt. # Test-thread which runs a ZK Failover Controller corresponding to a given dummy service - it's not a dummy service in this case. HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3089) Move FSDatasetInterface and other related classes/interfaces to a package
[ https://issues.apache.org/jira/browse/HDFS-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238928#comment-13238928 ] Hudson commented on HDFS-3089: -- Integrated in Hadoop-Mapreduce-0.23-Commit #740 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/740/]) svn merge -c 1305603 from trunk for HDFS-3089. Move FSDatasetInterface and the related classes to a package. (Revision 1305606) Revert 1305594 for HDFS-3089. (Revision 1305600) svn merge -c 1305590 from trunk for HDFS-3089. Move FSDatasetInterface and the related classes to a package. (Revision 1305594) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305606 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockVolumeChoosingPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataBlockScanner.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipelineInterface.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/RollingLogs.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/RoundRobinVolumesPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RollingLogs.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RoundRobinVolumeChoosingPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/VolumeChoosingPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DataNodeCluster.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestRoundRobinVolumesPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/TestRoundRobinVolumeChoosingPolicy.java szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305600 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs *
[jira] [Commented] (HDFS-3089) Move FSDatasetInterface and other related classes/interfaces to a package
[ https://issues.apache.org/jira/browse/HDFS-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238927#comment-13238927 ] Hudson commented on HDFS-3089: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1943 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1943/]) (recommit) HDFS-3089. Move FSDatasetInterface and the related classes to a package. (Revision 1305603) Revert 1305590 for HDFS-3089. (Revision 1305598) HDFS-3089. Move FSDatasetInterface and the related classes to a package. (Revision 1305590) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305603 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockVolumeChoosingPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataBlockScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipelineInterface.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/RollingLogs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/RoundRobinVolumesPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RollingLogs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RoundRobinVolumeChoosingPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/VolumeChoosingPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DataNodeCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestRoundRobinVolumesPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/TestRoundRobinVolumeChoosingPolicy.java szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305598 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockVolumeChoosingPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataBlockScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java *
[jira] [Updated] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2185: -- Attachment: hdfs-2185.txt New patch addresses all of the above except for #3. I don't think there's a race condition here, since the NN itself always starts in standby mode, and the minicluster startup waits until it is fully active. Its associated ZKFC might be in the process of starting, but that's OK, because all of our assertions involve wait-loops until they reach the desired state. HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238942#comment-13238942 ] Aaron T. Myers commented on HDFS-2185: -- bq. I don't think there's a race condition here, since the NN itself always starts in standby mode, and the minicluster startup waits until it is fully active. Ah, I thought that MiniDFSCluster might return from waitActive while still in the INITIALIZING state. My mistake. I agree with you, then, that there's no race condition. Latest patch looks good to me. +1 pending Jenkins and the commit of the Common patch. HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3129) NetworkTopology: add test that getLeaf should check for invalid topologies
[ https://issues.apache.org/jira/browse/HDFS-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238950#comment-13238950 ] Hudson commented on HDFS-3129: -- Integrated in Hadoop-Common-0.23-Commit #731 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/731/]) svn merge -c 1305628 HDFS-3129 from trunk (Revision 1305630) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305630 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java NetworkTopology: add test that getLeaf should check for invalid topologies -- Key: HDFS-3129 URL: https://issues.apache.org/jira/browse/HDFS-3129 Project: Hadoop HDFS Issue Type: Test Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Fix For: 1.1.0, 0.23.3 Attachments: HDFS-3129-b1.001.patch, HDFS-3129.001.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3089) Move FSDatasetInterface and other related classes/interfaces to a package
[ https://issues.apache.org/jira/browse/HDFS-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238951#comment-13238951 ] Hudson commented on HDFS-3089: -- Integrated in Hadoop-Common-0.23-Commit #731 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/731/]) svn merge -c 1305603 from trunk for HDFS-3089. Move FSDatasetInterface and the related classes to a package. (Revision 1305606) Revert 1305594 for HDFS-3089. (Revision 1305600) svn merge -c 1305590 from trunk for HDFS-3089. Move FSDatasetInterface and the related classes to a package. (Revision 1305594) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305606 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockVolumeChoosingPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataBlockScanner.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipelineInterface.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/RollingLogs.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/RoundRobinVolumesPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RollingLogs.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RoundRobinVolumeChoosingPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/VolumeChoosingPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DataNodeCluster.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestRoundRobinVolumesPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/TestRoundRobinVolumeChoosingPolicy.java szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305600 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs *
[jira] [Commented] (HDFS-2413) Add public APIs for safemode
[ https://issues.apache.org/jira/browse/HDFS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238960#comment-13238960 ] Hudson commented on HDFS-2413: -- Integrated in Hadoop-Common-trunk-Commit #1932 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1932/]) HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. Contributed by harsh (Revision 1305632) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305632 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java Add public APIs for safemode Key: HDFS-2413 URL: https://issues.apache.org/jira/browse/HDFS-2413 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Fix For: 0.24.0, 0.23.3 Attachments: HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3089) Move FSDatasetInterface and other related classes/interfaces to a package
[ https://issues.apache.org/jira/browse/HDFS-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238959#comment-13238959 ] Hudson commented on HDFS-3089: -- Integrated in Hadoop-Common-trunk-Commit #1932 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1932/]) (recommit) HDFS-3089. Move FSDatasetInterface and the related classes to a package. (Revision 1305603) Revert 1305590 for HDFS-3089. (Revision 1305598) HDFS-3089. Move FSDatasetInterface and the related classes to a package. (Revision 1305590) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305603 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockVolumeChoosingPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataBlockScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipelineInterface.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/RollingLogs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/RoundRobinVolumesPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RollingLogs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RoundRobinVolumeChoosingPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/VolumeChoosingPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DataNodeCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestRoundRobinVolumesPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/TestRoundRobinVolumeChoosingPolicy.java szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305598 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockVolumeChoosingPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataBlockScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java *
[jira] [Commented] (HDFS-3129) NetworkTopology: add test that getLeaf should check for invalid topologies
[ https://issues.apache.org/jira/browse/HDFS-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238978#comment-13238978 ] Hudson commented on HDFS-3129: -- Integrated in Hadoop-Hdfs-0.23-Commit #721 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/721/]) svn merge -c 1305628 HDFS-3129 from trunk (Revision 1305630) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305630 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java NetworkTopology: add test that getLeaf should check for invalid topologies -- Key: HDFS-3129 URL: https://issues.apache.org/jira/browse/HDFS-3129 Project: Hadoop HDFS Issue Type: Test Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Fix For: 1.1.0, 0.23.3 Attachments: HDFS-3129-b1.001.patch, HDFS-3129.001.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2413) Add public APIs for safemode
[ https://issues.apache.org/jira/browse/HDFS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238980#comment-13238980 ] Hudson commented on HDFS-2413: -- Integrated in Hadoop-Hdfs-0.23-Commit #721 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/721/]) svn merge -c 1305632 from trunk for HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. (Revision 1305633) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305633 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java Add public APIs for safemode Key: HDFS-2413 URL: https://issues.apache.org/jira/browse/HDFS-2413 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Fix For: 0.24.0, 0.23.3 Attachments: HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3089) Move FSDatasetInterface and other related classes/interfaces to a package
[ https://issues.apache.org/jira/browse/HDFS-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238979#comment-13238979 ] Hudson commented on HDFS-3089: -- Integrated in Hadoop-Hdfs-0.23-Commit #721 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/721/]) svn merge -c 1305603 from trunk for HDFS-3089. Move FSDatasetInterface and the related classes to a package. (Revision 1305606) Revert 1305594 for HDFS-3089. (Revision 1305600) svn merge -c 1305590 from trunk for HDFS-3089. Move FSDatasetInterface and the related classes to a package. (Revision 1305594) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305606 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockVolumeChoosingPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataBlockScanner.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipelineInterface.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/RollingLogs.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/RoundRobinVolumesPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RollingLogs.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RoundRobinVolumeChoosingPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/VolumeChoosingPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DataNodeCluster.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestRoundRobinVolumesPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/TestRoundRobinVolumeChoosingPolicy.java szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305600 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs *
[jira] [Commented] (HDFS-3129) NetworkTopology: add test that getLeaf should check for invalid topologies
[ https://issues.apache.org/jira/browse/HDFS-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238983#comment-13238983 ] Hudson commented on HDFS-3129: -- Integrated in Hadoop-Hdfs-trunk-Commit #2007 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2007/]) Fix CHANGES.txt for HDFS-3129 (Revision 1305631) HDFS-3129. NetworkTopology: add test that getLeaf should check for invalid topologies. Contributed by Colin Patrick McCabe (Revision 1305628) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305631 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305628 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java NetworkTopology: add test that getLeaf should check for invalid topologies -- Key: HDFS-3129 URL: https://issues.apache.org/jira/browse/HDFS-3129 Project: Hadoop HDFS Issue Type: Test Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Fix For: 1.1.0, 0.23.3 Attachments: HDFS-3129-b1.001.patch, HDFS-3129.001.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3154) Add a notion of immutable/mutable files
[ https://issues.apache.org/jira/browse/HDFS-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238988#comment-13238988 ] Hari Mankude commented on HDFS-3154: bq. You should be able to make a copy of an immutable file that takes up no extra space, but can be appended to or truncated. The immutable file and the copy would share immutable blocks. RO snaps provide the immutable capability for the description provided with this comment. Add a notion of immutable/mutable files --- Key: HDFS-3154 URL: https://issues.apache.org/jira/browse/HDFS-3154 Project: Hadoop HDFS Issue Type: New Feature Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE The notion of immutable file is useful since it lets the system and tools optimize certain things as discussed in [this email thread|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201203.mbox/%3CCAPn_vTuZomPmBTypP8_1xTr49Sj0fy7Mjhik4DbcAA+BLH53=g...@mail.gmail.com%3E]. Also, many applications require only immutable files. Here is a proposal: - Immutable files means that the file content is immutable. Operations such as append and truncate that change the file content are not allowed to act on immutable files. However, the meta data such as replication and permission of an immutable file can be updated. Immutable files can also be deleted or renamed. - Users have to pass immutable/mutable as a flag in file creation. This is an unmodifiable property of the created file. - If users want to change the data in an immutable file, the file could be copied to another file which is created as mutable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2413) Add public APIs for safemode
[ https://issues.apache.org/jira/browse/HDFS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238985#comment-13238985 ] Hudson commented on HDFS-2413: -- Integrated in Hadoop-Hdfs-trunk-Commit #2007 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2007/]) HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. Contributed by harsh (Revision 1305632) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305632 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java Add public APIs for safemode Key: HDFS-2413 URL: https://issues.apache.org/jira/browse/HDFS-2413 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Fix For: 0.24.0, 0.23.3 Attachments: HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3089) Move FSDatasetInterface and other related classes/interfaces to a package
[ https://issues.apache.org/jira/browse/HDFS-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238984#comment-13238984 ] Hudson commented on HDFS-3089: -- Integrated in Hadoop-Hdfs-trunk-Commit #2007 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2007/]) (recommit) HDFS-3089. Move FSDatasetInterface and the related classes to a package. (Revision 1305603) Revert 1305590 for HDFS-3089. (Revision 1305598) HDFS-3089. Move FSDatasetInterface and the related classes to a package. (Revision 1305590) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305603 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockVolumeChoosingPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataBlockScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipelineInterface.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/RollingLogs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/RoundRobinVolumesPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RollingLogs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/RoundRobinVolumeChoosingPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/VolumeChoosingPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DataNodeCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestRoundRobinVolumesPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/TestRoundRobinVolumeChoosingPolicy.java szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305598 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockVolumeChoosingPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataBlockScanner.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java *
[jira] [Commented] (HDFS-3129) NetworkTopology: add test that getLeaf should check for invalid topologies
[ https://issues.apache.org/jira/browse/HDFS-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238996#comment-13238996 ] Hudson commented on HDFS-3129: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1944 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1944/]) Fix CHANGES.txt for HDFS-3129 (Revision 1305631) HDFS-3129. NetworkTopology: add test that getLeaf should check for invalid topologies. Contributed by Colin Patrick McCabe (Revision 1305628) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305631 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305628 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java NetworkTopology: add test that getLeaf should check for invalid topologies -- Key: HDFS-3129 URL: https://issues.apache.org/jira/browse/HDFS-3129 Project: Hadoop HDFS Issue Type: Test Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Fix For: 1.1.0, 0.23.3 Attachments: HDFS-3129-b1.001.patch, HDFS-3129.001.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2413) Add public APIs for safemode
[ https://issues.apache.org/jira/browse/HDFS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238997#comment-13238997 ] Hudson commented on HDFS-2413: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1944 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1944/]) HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. Contributed by harsh (Revision 1305632) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305632 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java Add public APIs for safemode Key: HDFS-2413 URL: https://issues.apache.org/jira/browse/HDFS-2413 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Fix For: 0.24.0, 0.23.3 Attachments: HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3129) NetworkTopology: add test that getLeaf should check for invalid topologies
[ https://issues.apache.org/jira/browse/HDFS-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238994#comment-13238994 ] Hudson commented on HDFS-3129: -- Integrated in Hadoop-Mapreduce-0.23-Commit #741 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/741/]) svn merge -c 1305628 HDFS-3129 from trunk (Revision 1305630) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305630 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java NetworkTopology: add test that getLeaf should check for invalid topologies -- Key: HDFS-3129 URL: https://issues.apache.org/jira/browse/HDFS-3129 Project: Hadoop HDFS Issue Type: Test Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Fix For: 1.1.0, 0.23.3 Attachments: HDFS-3129-b1.001.patch, HDFS-3129.001.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2413) Add public APIs for safemode
[ https://issues.apache.org/jira/browse/HDFS-2413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238995#comment-13238995 ] Hudson commented on HDFS-2413: -- Integrated in Hadoop-Mapreduce-0.23-Commit #741 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/741/]) svn merge -c 1305632 from trunk for HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. (Revision 1305633) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305633 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java Add public APIs for safemode Key: HDFS-2413 URL: https://issues.apache.org/jira/browse/HDFS-2413 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Fix For: 0.24.0, 0.23.3 Attachments: HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch, HDFS-2413.patch Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2185: -- Target Version/s: 0.24.0, 0.23.3 (was: HA branch (HDFS-1623)) Affects Version/s: (was: HA branch (HDFS-1623)) 0.23.3 0.24.0 HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2185: -- Status: Patch Available (was: Open) HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239014#comment-13239014 ] Sanjay Radia commented on HDFS-2185: Todd, thanks for posting the design doc. I am reading it. I noticed that there isn't a state transition diagram. I noticed that Bikas has posted a state transition diagram - not sure if you are using the same or another one. If not could you add one please - this part of the system is fairly subtle and a state transition diagram would help. HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239024#comment-13239024 ] Todd Lipcon commented on HDFS-2185: --- Hi Sanjay. I'll work on adding a state machine diagram to the design doc. My graphviz skills are rusty, but should have something by tonight. HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3131) Improve TestStorageRestore
[ https://issues.apache.org/jira/browse/HDFS-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239031#comment-13239031 ] Brandon Li commented on HDFS-3131: -- Thank you, Aaron! Improve TestStorageRestore -- Key: HDFS-3131 URL: https://issues.apache.org/jira/browse/HDFS-3131 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.24.0, 1.1.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Brandon Li Priority: Minor Labels: newbie Attachments: HDFS-3131.branch-1.patch, HDFS-3131.patch, HDFS-3131.patch Aaron has the following comments on TestStorageRestore in HDFS-3127. # removeStorageAccess, restoreAccess, and numStorageDirs can all be made private # numStorageDirs can be made static # Rather than do set(Readable/Executable/Writable), use FileUtil.chmod(...). # Please put the contents of the test in a try/finally, with the calls to shutdown the cluster and the 2NN in the finally block. # Some lines are over 80 chars. # No need for the numDatanodes variable - it's only used in one place. # Instead of xwr use rwx, which I think is a more common way of describing permissions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3131) Improve TestStorageRestore
[ https://issues.apache.org/jira/browse/HDFS-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3131: - Resolution: Fixed Fix Version/s: 1.1.0 0.24.0 Target Version/s: 0.24.0, 1.1.0 (was: 1.1.0, 0.24.0) Release Note: (was: added test case to make sure that restore failure doesn't stop checkpoint process) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-1. Thanks a lot for the contribution, Brandon. Improve TestStorageRestore -- Key: HDFS-3131 URL: https://issues.apache.org/jira/browse/HDFS-3131 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.24.0, 1.1.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Brandon Li Priority: Minor Labels: newbie Fix For: 0.24.0, 1.1.0 Attachments: HDFS-3131.branch-1.patch, HDFS-3131.patch, HDFS-3131.patch Aaron has the following comments on TestStorageRestore in HDFS-3127. # removeStorageAccess, restoreAccess, and numStorageDirs can all be made private # numStorageDirs can be made static # Rather than do set(Readable/Executable/Writable), use FileUtil.chmod(...). # Please put the contents of the test in a try/finally, with the calls to shutdown the cluster and the 2NN in the finally block. # Some lines are over 80 chars. # No need for the numDatanodes variable - it's only used in one place. # Instead of xwr use rwx, which I think is a more common way of describing permissions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239053#comment-13239053 ] Aaron T. Myers commented on HDFS-2185: -- The latest changes look good to me. +1 pending Jenkins. It also occurs to me that this change warrants some updates to some user docs, notably ClusterSetup.apt.vm (to mention HADOOP_ZKFC_OPTS) and HDFSHighAvailability.apt.vm. These changes should be a separate JIRA, though, since those docs are not under hadoop-hdfs-project. HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3131) Improve TestStorageRestore
[ https://issues.apache.org/jira/browse/HDFS-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239068#comment-13239068 ] Hudson commented on HDFS-3131: -- Integrated in Hadoop-Common-trunk-Commit #1933 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1933/]) HDFS-3131. Improve TestStorageRestore. Contributed by Brandon Li. (Revision 1305688) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305688 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStorageRestore.java Improve TestStorageRestore -- Key: HDFS-3131 URL: https://issues.apache.org/jira/browse/HDFS-3131 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.24.0, 1.1.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Brandon Li Priority: Minor Labels: newbie Fix For: 0.24.0, 1.1.0 Attachments: HDFS-3131.branch-1.patch, HDFS-3131.patch, HDFS-3131.patch Aaron has the following comments on TestStorageRestore in HDFS-3127. # removeStorageAccess, restoreAccess, and numStorageDirs can all be made private # numStorageDirs can be made static # Rather than do set(Readable/Executable/Writable), use FileUtil.chmod(...). # Please put the contents of the test in a try/finally, with the calls to shutdown the cluster and the 2NN in the finally block. # Some lines are over 80 chars. # No need for the numDatanodes variable - it's only used in one place. # Instead of xwr use rwx, which I think is a more common way of describing permissions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3131) Improve TestStorageRestore
[ https://issues.apache.org/jira/browse/HDFS-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239081#comment-13239081 ] Hudson commented on HDFS-3131: -- Integrated in Hadoop-Hdfs-trunk-Commit #2008 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2008/]) HDFS-3131. Improve TestStorageRestore. Contributed by Brandon Li. (Revision 1305688) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305688 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStorageRestore.java Improve TestStorageRestore -- Key: HDFS-3131 URL: https://issues.apache.org/jira/browse/HDFS-3131 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.24.0, 1.1.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Brandon Li Priority: Minor Labels: newbie Fix For: 0.24.0, 1.1.0 Attachments: HDFS-3131.branch-1.patch, HDFS-3131.patch, HDFS-3131.patch Aaron has the following comments on TestStorageRestore in HDFS-3127. # removeStorageAccess, restoreAccess, and numStorageDirs can all be made private # numStorageDirs can be made static # Rather than do set(Readable/Executable/Writable), use FileUtil.chmod(...). # Please put the contents of the test in a try/finally, with the calls to shutdown the cluster and the 2NN in the finally block. # Some lines are over 80 chars. # No need for the numDatanodes variable - it's only used in one place. # Instead of xwr use rwx, which I think is a more common way of describing permissions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3131) Improve TestStorageRestore
[ https://issues.apache.org/jira/browse/HDFS-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239085#comment-13239085 ] Hadoop QA commented on HDFS-3131: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520018/HDFS-3131.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestGetBlocks org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings org.apache.hadoop.hdfs.server.namenode.TestStartup +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2101//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2101//console This message is automatically generated. Improve TestStorageRestore -- Key: HDFS-3131 URL: https://issues.apache.org/jira/browse/HDFS-3131 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.24.0, 1.1.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Brandon Li Priority: Minor Labels: newbie Fix For: 0.24.0, 1.1.0 Attachments: HDFS-3131.branch-1.patch, HDFS-3131.patch, HDFS-3131.patch Aaron has the following comments on TestStorageRestore in HDFS-3127. # removeStorageAccess, restoreAccess, and numStorageDirs can all be made private # numStorageDirs can be made static # Rather than do set(Readable/Executable/Writable), use FileUtil.chmod(...). # Please put the contents of the test in a try/finally, with the calls to shutdown the cluster and the 2NN in the finally block. # Some lines are over 80 chars. # No need for the numDatanodes variable - it's only used in one place. # Instead of xwr use rwx, which I think is a more common way of describing permissions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2185: -- Attachment: zkfc-design.tex New revision of the design doc: - includes a diagram of the state machine - amends the description of the 'ZooKeeper crash' scenario. I had a mistake in my understanding of ZK's behavior before - now clarified. - adds mention of the INITIALIZING state per atm's feedback in another JIRA I also pushed the latex source for the design doc here: https://github.com/toddlipcon/zkfc-design HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf, zkfc-design.pdf, zkfc-design.tex This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2185: -- Attachment: zkfc-design.pdf Meant to attach the pdf instead of the .tex file. HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf, zkfc-design.pdf, zkfc-design.tex This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3125) Add a service that enables JournalDaemon
[ https://issues.apache.org/jira/browse/HDFS-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239101#comment-13239101 ] Hadoop QA commented on HDFS-3125: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520014/HDFS-3125.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestGetBlocks org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2102//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2102//console This message is automatically generated. Add a service that enables JournalDaemon Key: HDFS-3125 URL: https://issues.apache.org/jira/browse/HDFS-3125 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-3125.patch, HDFS-3125.patch, HDFS-3125.patch, HDFS-3125.patch In this subtask, I plan to add JournalService. It will provide the following functionality: # Starts RPC server with JournalProtocolService or uses the RPC server provided and add JournalProtocol service. # Registers with the namenode. # Receives JournalProtocol related requests and hands it to over to a listener. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3131) Improve TestStorageRestore
[ https://issues.apache.org/jira/browse/HDFS-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239100#comment-13239100 ] Hudson commented on HDFS-3131: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1946 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1946/]) HDFS-3131. Improve TestStorageRestore. Contributed by Brandon Li. (Revision 1305688) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305688 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStorageRestore.java Improve TestStorageRestore -- Key: HDFS-3131 URL: https://issues.apache.org/jira/browse/HDFS-3131 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.24.0, 1.1.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Brandon Li Priority: Minor Labels: newbie Fix For: 0.24.0, 1.1.0 Attachments: HDFS-3131.branch-1.patch, HDFS-3131.patch, HDFS-3131.patch Aaron has the following comments on TestStorageRestore in HDFS-3127. # removeStorageAccess, restoreAccess, and numStorageDirs can all be made private # numStorageDirs can be made static # Rather than do set(Readable/Executable/Writable), use FileUtil.chmod(...). # Please put the contents of the test in a try/finally, with the calls to shutdown the cluster and the 2NN in the finally block. # Some lines are over 80 chars. # No need for the numDatanodes variable - it's only used in one place. # Instead of xwr use rwx, which I think is a more common way of describing permissions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239103#comment-13239103 ] Hadoop QA commented on HDFS-2185: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520047/zkfc-design.pdf against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2104//console This message is automatically generated. HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf, zkfc-design.pdf, zkfc-design.tex This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3125) Add a service that enables JournalDaemon
[ https://issues.apache.org/jira/browse/HDFS-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239104#comment-13239104 ] Tsz Wo (Nicholas), SZE commented on HDFS-3125: -- +1 the latest patch look goods. Add a service that enables JournalDaemon Key: HDFS-3125 URL: https://issues.apache.org/jira/browse/HDFS-3125 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-3125.patch, HDFS-3125.patch, HDFS-3125.patch, HDFS-3125.patch In this subtask, I plan to add JournalService. It will provide the following functionality: # Starts RPC server with JournalProtocolService or uses the RPC server provided and add JournalProtocol service. # Registers with the namenode. # Receives JournalProtocol related requests and hands it to over to a listener. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2185: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520047/zkfc-design.pdf against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2104//console This message is automatically generated.) HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf, zkfc-design.pdf, zkfc-design.tex This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239113#comment-13239113 ] Hadoop QA commented on HDFS-2185: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520036/hdfs-2185.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.namenode.ha.TestDFSZKFailoverController org.apache.hadoop.hdfs.TestGetBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2103//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2103//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2103//console This message is automatically generated. HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf, zkfc-design.pdf, zkfc-design.tex This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2185: -- Target Version/s: 0.24.0, 0.23.3 (was: 0.23.3, 0.24.0) Status: Open (was: Patch Available) Canceling patch to address test failure. Looks like it may be a race. I'll loop the test to try to uncover it. HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf, zkfc-design.pdf, zkfc-design.tex This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3154) Add a notion of immutable/mutable files
[ https://issues.apache.org/jira/browse/HDFS-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239126#comment-13239126 ] Tsz Wo (Nicholas), SZE commented on HDFS-3154: -- Can you list the concrete benefits you think would come from having some, but not all, files be immutable? ... The main benefit is caching. Another benefit is to protect the files. It avoid accidentally append/truncate on immutable files. I did not know this was common until recently [from Milind|https://issues.apache.org/jira/browse/HDFS-3107?focusedCommentId=13233609page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13233609]. Add a notion of immutable/mutable files --- Key: HDFS-3154 URL: https://issues.apache.org/jira/browse/HDFS-3154 Project: Hadoop HDFS Issue Type: New Feature Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE The notion of immutable file is useful since it lets the system and tools optimize certain things as discussed in [this email thread|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201203.mbox/%3CCAPn_vTuZomPmBTypP8_1xTr49Sj0fy7Mjhik4DbcAA+BLH53=g...@mail.gmail.com%3E]. Also, many applications require only immutable files. Here is a proposal: - Immutable files means that the file content is immutable. Operations such as append and truncate that change the file content are not allowed to act on immutable files. However, the meta data such as replication and permission of an immutable file can be updated. Immutable files can also be deleted or renamed. - Users have to pass immutable/mutable as a flag in file creation. This is an unmodifiable property of the created file. - If users want to change the data in an immutable file, the file could be copied to another file which is created as mutable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3125) Add a service that enables JournalDaemon
[ https://issues.apache.org/jira/browse/HDFS-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-3125: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed the patch. Add a service that enables JournalDaemon Key: HDFS-3125 URL: https://issues.apache.org/jira/browse/HDFS-3125 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-3125.patch, HDFS-3125.patch, HDFS-3125.patch, HDFS-3125.patch In this subtask, I plan to add JournalService. It will provide the following functionality: # Starts RPC server with JournalProtocolService or uses the RPC server provided and add JournalProtocol service. # Registers with the namenode. # Receives JournalProtocol related requests and hands it to over to a listener. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3125) Add a service that enables JournalDaemon
[ https://issues.apache.org/jira/browse/HDFS-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239160#comment-13239160 ] Hudson commented on HDFS-3125: -- Integrated in Hadoop-Common-trunk-Commit #1935 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1935/]) HDFS-3125. Add JournalService to enable Journal Daemon. Contributed by Suresh Srinivas. (Revision 1305726) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305726 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestJournalService.java Add a service that enables JournalDaemon Key: HDFS-3125 URL: https://issues.apache.org/jira/browse/HDFS-3125 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-3125.patch, HDFS-3125.patch, HDFS-3125.patch, HDFS-3125.patch In this subtask, I plan to add JournalService. It will provide the following functionality: # Starts RPC server with JournalProtocolService or uses the RPC server provided and add JournalProtocol service. # Registers with the namenode. # Receives JournalProtocol related requests and hands it to over to a listener. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3125) Add a service that enables JournalDaemon
[ https://issues.apache.org/jira/browse/HDFS-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239161#comment-13239161 ] Hudson commented on HDFS-3125: -- Integrated in Hadoop-Hdfs-trunk-Commit #2010 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2010/]) HDFS-3125. Add JournalService to enable Journal Daemon. Contributed by Suresh Srinivas. (Revision 1305726) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305726 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestJournalService.java Add a service that enables JournalDaemon Key: HDFS-3125 URL: https://issues.apache.org/jira/browse/HDFS-3125 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-3125.patch, HDFS-3125.patch, HDFS-3125.patch, HDFS-3125.patch In this subtask, I plan to add JournalService. It will provide the following functionality: # Starts RPC server with JournalProtocolService or uses the RPC server provided and add JournalProtocol service. # Registers with the namenode. # Receives JournalProtocol related requests and hands it to over to a listener. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239183#comment-13239183 ] Todd Lipcon commented on HDFS-2185: --- I think the race we hit was due to HADOOP-8220. Let's get that one committed and then try resubmitting this one. With that bug fixed in my local tree, I was able to loop the new test for a number of iterations without any failures. HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Eli Collins Assignee: Todd Lipcon Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf, zkfc-design.pdf, zkfc-design.tex This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3125) Add a service that enables JournalDaemon
[ https://issues.apache.org/jira/browse/HDFS-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239188#comment-13239188 ] Hudson commented on HDFS-3125: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1948 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1948/]) HDFS-3125. Add JournalService to enable Journal Daemon. Contributed by Suresh Srinivas. (Revision 1305726) Result = ABORTED suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1305726 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestJournalService.java Add a service that enables JournalDaemon Key: HDFS-3125 URL: https://issues.apache.org/jira/browse/HDFS-3125 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-3125.patch, HDFS-3125.patch, HDFS-3125.patch, HDFS-3125.patch In this subtask, I plan to add JournalService. It will provide the following functionality: # Starts RPC server with JournalProtocolService or uses the RPC server provided and add JournalProtocol service. # Registers with the namenode. # Receives JournalProtocol related requests and hands it to over to a listener. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
