[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[ https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435045#comment-13435045 ] Hudson commented on HDFS-3765: -- Integrated in Hadoop-Hdfs-trunk #1135 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1135/]) HDFS-3765. namenode -initializeSharedEdits should be able to initialize all shared storages. Contributed by Vinay and Todd Lipcon. (Revision 1373061) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1373061 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestInitializeSharedEdits.java Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages --- Key: HDFS-3765 URL: https://issues.apache.org/jira/browse/HDFS-3765 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 2.1.0-alpha, 3.0.0 Reporter: Vinay Assignee: Vinay Fix For: 3.0.0, 2.2.0-alpha Attachments: hdfs-3765-branch-2.txt, HDFS-3765.patch, HDFS-3765.patch, HDFS-3765.patch, hdfs-3765.txt, hdfs-3765.txt, hdfs-3765.txt Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits files to file schema based shared storages when moving cluster from Non-HA environment to HA enabled environment. This Jira focuses on the following * Generalizing the logic of copying the edits to new shared storage so that any schema based shared storage can initialized for HA cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[ https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435074#comment-13435074 ] Hudson commented on HDFS-3765: -- Integrated in Hadoop-Mapreduce-trunk #1167 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1167/]) HDFS-3765. namenode -initializeSharedEdits should be able to initialize all shared storages. Contributed by Vinay and Todd Lipcon. (Revision 1373061) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1373061 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestInitializeSharedEdits.java Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages --- Key: HDFS-3765 URL: https://issues.apache.org/jira/browse/HDFS-3765 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 2.1.0-alpha, 3.0.0 Reporter: Vinay Assignee: Vinay Fix For: 3.0.0, 2.2.0-alpha Attachments: hdfs-3765-branch-2.txt, HDFS-3765.patch, HDFS-3765.patch, HDFS-3765.patch, hdfs-3765.txt, hdfs-3765.txt, hdfs-3765.txt Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits files to file schema based shared storages when moving cluster from Non-HA environment to HA enabled environment. This Jira focuses on the following * Generalizing the logic of copying the edits to new shared storage so that any schema based shared storage can initialized for HA cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[ https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13434321#comment-13434321 ] Aaron T. Myers commented on HDFS-3765: -- +1, the latest patch looks good to me. Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages --- Key: HDFS-3765 URL: https://issues.apache.org/jira/browse/HDFS-3765 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 2.1.0-alpha, 3.0.0 Reporter: Vinay Assignee: Vinay Attachments: HDFS-3765.patch, HDFS-3765.patch, HDFS-3765.patch, hdfs-3765.txt, hdfs-3765.txt, hdfs-3765.txt Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits files to file schema based shared storages when moving cluster from Non-HA environment to HA enabled environment. This Jira focuses on the following * Generalizing the logic of copying the edits to new shared storage so that any schema based shared storage can initialized for HA cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[ https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13434674#comment-13434674 ] Hudson commented on HDFS-3765: -- Integrated in Hadoop-Common-trunk-Commit #2577 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2577/]) HDFS-3765. namenode -initializeSharedEdits should be able to initialize all shared storages. Contributed by Vinay and Todd Lipcon. (Revision 1373061) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1373061 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestInitializeSharedEdits.java Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages --- Key: HDFS-3765 URL: https://issues.apache.org/jira/browse/HDFS-3765 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 2.1.0-alpha, 3.0.0 Reporter: Vinay Assignee: Vinay Fix For: 3.0.0, 2.2.0-alpha Attachments: hdfs-3765-branch-2.txt, HDFS-3765.patch, HDFS-3765.patch, HDFS-3765.patch, hdfs-3765.txt, hdfs-3765.txt, hdfs-3765.txt Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits files to file schema based shared storages when moving cluster from Non-HA environment to HA enabled environment. This Jira focuses on the following * Generalizing the logic of copying the edits to new shared storage so that any schema based shared storage can initialized for HA cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[ https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13434679#comment-13434679 ] Hudson commented on HDFS-3765: -- Integrated in Hadoop-Hdfs-trunk-Commit #2642 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2642/]) HDFS-3765. namenode -initializeSharedEdits should be able to initialize all shared storages. Contributed by Vinay and Todd Lipcon. (Revision 1373061) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1373061 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperAsHASharedDir.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestInitializeSharedEdits.java Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages --- Key: HDFS-3765 URL: https://issues.apache.org/jira/browse/HDFS-3765 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 2.1.0-alpha, 3.0.0 Reporter: Vinay Assignee: Vinay Fix For: 3.0.0, 2.2.0-alpha Attachments: hdfs-3765-branch-2.txt, HDFS-3765.patch, HDFS-3765.patch, HDFS-3765.patch, hdfs-3765.txt, hdfs-3765.txt, hdfs-3765.txt Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits files to file schema based shared storages when moving cluster from Non-HA environment to HA enabled environment. This Jira focuses on the following * Generalizing the logic of copying the edits to new shared storage so that any schema based shared storage can initialized for HA cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[ https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13433797#comment-13433797 ] Hadoop QA commented on HDFS-3765: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540746/hdfs-3765.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2995//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2995//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2995//console This message is automatically generated. Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages --- Key: HDFS-3765 URL: https://issues.apache.org/jira/browse/HDFS-3765 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 2.1.0-alpha, 3.0.0 Reporter: Vinay Assignee: Vinay Attachments: HDFS-3765.patch, HDFS-3765.patch, HDFS-3765.patch, hdfs-3765.txt, hdfs-3765.txt, hdfs-3765.txt Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits files to file schema based shared storages when moving cluster from Non-HA environment to HA enabled environment. This Jira focuses on the following * Generalizing the logic of copying the edits to new shared storage so that any schema based shared storage can initialized for HA cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[ https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13433800#comment-13433800 ] Todd Lipcon commented on HDFS-3765: --- The findbugs warnings are from HDFS-3695. They have already been fixed in HDFS-3792 (committed just before this QA report) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages --- Key: HDFS-3765 URL: https://issues.apache.org/jira/browse/HDFS-3765 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 2.1.0-alpha, 3.0.0 Reporter: Vinay Assignee: Vinay Attachments: HDFS-3765.patch, HDFS-3765.patch, HDFS-3765.patch, hdfs-3765.txt, hdfs-3765.txt, hdfs-3765.txt Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits files to file schema based shared storages when moving cluster from Non-HA environment to HA enabled environment. This Jira focuses on the following * Generalizing the logic of copying the edits to new shared storage so that any schema based shared storage can initialized for HA cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[
https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13431617#comment-13431617
]
Vinay commented on HDFS-3765:
-
{quote}NameNode -initializeSharedEdits (the command is not generally typed in
all caps)
capitalize BKJM
no need for @throws Exception since it's redundant with the signature
itself{quote}
Ok. I will do it.
{quote}Calling logSync on every edit will make this take a really long time if
there are a non-trivial number of edits to copy. Instead, I think you can avoid
calling logSync, and instead rely on the automatic sync feature of the edit
streams to trigger a logSync when the buffer grows too large.{quote}
Ok. I will use automatic sync
{quote}Otherwise the END_LOG_SEGMENT transaction doesn't end up logged in
shared edits, and you end up with a one-op gap in between every segment.{quote}
Here {{newSharedEditLog.endCurrentLogSegment(true)}} call itself will log
END_LOG_SEGMENT based on the boolean we passed.
Here is the code inside {{FSEditLog.endCurrentLogSegment(boolean)}}
{code}if (writeEndTxn) {
logEdit(LogSegmentOp.getInstance(cache.get(),
FSEditLogOpCodes.OP_END_LOG_SEGMENT));
logSync();
}{code}
No need to explicitly write END_LOG_SEGMENT op.
{quote}Oh, and please remove the now-unused imports in NameNode.java as
well.{quote}
Sure.
{quote}Here's a rev with some of the above changes. Also fixed
TestBookKeeperAsHASharedDir to properly reset the ExitException in a @Before
method.{quote}
Here I didn't find method {{ExitUtil.resetFirstExitException()}} in
ExitUtil.java
I had run the same Unit tests, all had passed here. Anyway I will run unit
tests one more time, before posting the patch.
Namenode INITIALIZESHAREDEDITS should be able to initialize all shared
storages
---
Key: HDFS-3765
URL: https://issues.apache.org/jira/browse/HDFS-3765
Project: Hadoop HDFS
Issue Type: Improvement
Components: ha
Affects Versions: 2.1.0-alpha, 3.0.0
Reporter: Vinay
Assignee: Vinay
Attachments: HDFS-3765.patch, HDFS-3765.patch, hdfs-3765.txt
Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits
files to file schema based shared storages when moving cluster from Non-HA
environment to HA enabled environment.
This Jira focuses on the following
* Generalizing the logic of copying the edits to new shared storage so that
any schema based shared storage can initialized for HA cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[ https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13432006#comment-13432006 ] Hadoop QA commented on HDFS-3765: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540034/HDFS-3765.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal: org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics org.apache.hadoop.contrib.bkjournal.TestBookKeeperAsHASharedDir +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2982//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2982//console This message is automatically generated. Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages --- Key: HDFS-3765 URL: https://issues.apache.org/jira/browse/HDFS-3765 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 2.1.0-alpha, 3.0.0 Reporter: Vinay Assignee: Vinay Attachments: HDFS-3765.patch, HDFS-3765.patch, HDFS-3765.patch, hdfs-3765.txt Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits files to file schema based shared storages when moving cluster from Non-HA environment to HA enabled environment. This Jira focuses on the following * Generalizing the logic of copying the edits to new shared storage so that any schema based shared storage can initialized for HA cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[ https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13432334#comment-13432334 ] Hadoop QA commented on HDFS-3765: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540160/hdfs-3765.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2984//console This message is automatically generated. Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages --- Key: HDFS-3765 URL: https://issues.apache.org/jira/browse/HDFS-3765 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 2.1.0-alpha, 3.0.0 Reporter: Vinay Assignee: Vinay Attachments: HDFS-3765.patch, HDFS-3765.patch, HDFS-3765.patch, hdfs-3765.txt, hdfs-3765.txt Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits files to file schema based shared storages when moving cluster from Non-HA environment to HA enabled environment. This Jira focuses on the following * Generalizing the logic of copying the edits to new shared storage so that any schema based shared storage can initialized for HA cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[ https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13431373#comment-13431373 ] Hadoop QA commented on HDFS-3765: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539883/HDFS-3765.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal: org.apache.hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits org.apache.hadoop.contrib.bkjournal.TestBookKeeperAsHASharedDir +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2972//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2972//console This message is automatically generated. Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages --- Key: HDFS-3765 URL: https://issues.apache.org/jira/browse/HDFS-3765 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 2.1.0-alpha, 3.0.0 Reporter: Vinay Assignee: Vinay Attachments: HDFS-3765.patch, HDFS-3765.patch Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits files to file schema based shared storages when moving cluster from Non-HA environment to HA enabled environment. This Jira focuses on the following * Generalizing the logic of copying the edits to new shared storage so that any schema based shared storage can initialized for HA cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[
https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13431392#comment-13431392
]
Todd Lipcon commented on HDFS-3765:
---
Seems it's not passing its unit tests. I think the code needs to look like this
instead:
{code}
newSharedEditLog.logEdit(op);
if (op.opCode == FSEditLogOpCodes.OP_END_LOG_SEGMENT) {
newSharedEditLog.endCurrentLogSegment(true);
break;
}
{code}
Otherwise the END_LOG_SEGMENT transaction doesn't end up logged in shared
edits, and you end up with a one-op gap in between every segment.
Namenode INITIALIZESHAREDEDITS should be able to initialize all shared
storages
---
Key: HDFS-3765
URL: https://issues.apache.org/jira/browse/HDFS-3765
Project: Hadoop HDFS
Issue Type: Improvement
Components: ha
Affects Versions: 2.1.0-alpha, 3.0.0
Reporter: Vinay
Assignee: Vinay
Attachments: HDFS-3765.patch, HDFS-3765.patch
Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits
files to file schema based shared storages when moving cluster from Non-HA
environment to HA enabled environment.
This Jira focuses on the following
* Generalizing the logic of copying the edits to new shared storage so that
any schema based shared storage can initialized for HA cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[ https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13431393#comment-13431393 ] Todd Lipcon commented on HDFS-3765: --- Oh, and please remove the now-unused imports in NameNode.java as well. Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages --- Key: HDFS-3765 URL: https://issues.apache.org/jira/browse/HDFS-3765 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 2.1.0-alpha, 3.0.0 Reporter: Vinay Assignee: Vinay Attachments: HDFS-3765.patch, HDFS-3765.patch Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits files to file schema based shared storages when moving cluster from Non-HA environment to HA enabled environment. This Jira focuses on the following * Generalizing the logic of copying the edits to new shared storage so that any schema based shared storage can initialized for HA cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[ https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429363#comment-13429363 ] Todd Lipcon commented on HDFS-3765: --- Hey Vinay. Thanks a lot for doing this - it's been on my list but hadn't gotten to it yet. Do you plan to add a test case, perhaps against the BKJM implementation? I'll look at the code as soon as I can. Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages --- Key: HDFS-3765 URL: https://issues.apache.org/jira/browse/HDFS-3765 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 2.1.0-alpha, 3.0.0 Reporter: Vinay Assignee: Vinay Attachments: HDFS-3765.patch Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits files to file schema based shared storages when moving cluster from Non-HA environment to HA enabled environment. This Jira focuses on the following * Generalizing the logic of copying the edits to new shared storage so that any schema based shared storage can initialized for HA cluster. * Ability to Initialize new shared storage from existing shared storage when moving from One shared storage to another shared storage (Might be because of cost, performance, etc. For ex: Moving from NFS to BKJM/QJM). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[ https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429651#comment-13429651 ] Todd Lipcon commented on HDFS-3765: --- Code looks pretty reasonable. But I think we should separate this into two separate patches. I'm not 100% convinced the copy from one edits storage to another should be lumped in with initializeSharedEdits. Would you mind doing just the genericizing part in this JIRA and we can discuss the other use case separately? Also, please add a test which uses this new facility to initialize BKJM edits, if you don't mind. Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages --- Key: HDFS-3765 URL: https://issues.apache.org/jira/browse/HDFS-3765 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 2.1.0-alpha, 3.0.0 Reporter: Vinay Assignee: Vinay Attachments: HDFS-3765.patch Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits files to file schema based shared storages when moving cluster from Non-HA environment to HA enabled environment. This Jira focuses on the following * Generalizing the logic of copying the edits to new shared storage so that any schema based shared storage can initialized for HA cluster. * Ability to Initialize new shared storage from existing shared storage when moving from One shared storage to another shared storage (Might be because of cost, performance, etc. For ex: Moving from NFS to BKJM/QJM). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3765) Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
[
https://issues.apache.org/jira/browse/HDFS-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429912#comment-13429912
]
Vinay commented on HDFS-3765:
-
Thanks a lot Todd for taking a look.
{quote} I'm not 100% convinced the copy from one edits storage to another
should be lumped in with initializeSharedEdits{quote}
If you feel we can handle this in separate jira, then fine. I will concentrate
only on the genericizing part.
{quote}Also, please add a test which uses this new facility to initialize BKJM
edits, if you don't mind.{quote}
Sure, I will try to add a testcase in BKJM contrib module.
Namenode INITIALIZESHAREDEDITS should be able to initialize all shared
storages
---
Key: HDFS-3765
URL: https://issues.apache.org/jira/browse/HDFS-3765
Project: Hadoop HDFS
Issue Type: Improvement
Components: ha
Affects Versions: 2.1.0-alpha, 3.0.0
Reporter: Vinay
Assignee: Vinay
Attachments: HDFS-3765.patch
Currently, NameNode INITIALIZESHAREDEDITS provides ability to copy the edits
files to file schema based shared storages when moving cluster from Non-HA
environment to HA enabled environment.
This Jira focuses on the following
* Generalizing the logic of copying the edits to new shared storage so that
any schema based shared storage can initialized for HA cluster.
* Ability to Initialize new shared storage from existing shared storage when
moving from One shared storage to another shared storage (Might be because of
cost, performance, etc. For ex: Moving from NFS to BKJM/QJM).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
