[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14067546#comment-14067546
]
Hudson commented on HDFS-6597:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1836 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1836/])
HDFS-6597. Add a new option to NN upgrade to terminate the process after
upgrade on NN is completed. Contributed by Danilo Vunjak. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611723)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartupOptionUpgrade.java
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0, 2.5.0
>Reporter: Danilo Vunjak
>Assignee: Danilo Vunjak
> Fix For: 3.0.0, 2.6.0
>
> Attachments: HDFS-6597.04.patch, HDFS-6597.05.patch,
> HDFS-6597.06.patch, JIRA-HDFS-30.patch, JIRA-HDFS-6597.02.patch,
> JIRA-HDFS-6597.03.patch, JIRA-HDFS-6597.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14067522#comment-14067522
]
Hudson commented on HDFS-6597:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1809 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1809/])
HDFS-6597. Add a new option to NN upgrade to terminate the process after
upgrade on NN is completed. Contributed by Danilo Vunjak. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611723)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartupOptionUpgrade.java
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0, 2.5.0
>Reporter: Danilo Vunjak
>Assignee: Danilo Vunjak
> Fix For: 3.0.0, 2.6.0
>
> Attachments: HDFS-6597.04.patch, HDFS-6597.05.patch,
> HDFS-6597.06.patch, JIRA-HDFS-30.patch, JIRA-HDFS-6597.02.patch,
> JIRA-HDFS-6597.03.patch, JIRA-HDFS-6597.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14067473#comment-14067473
]
Hudson commented on HDFS-6597:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #617 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/617/])
HDFS-6597. Add a new option to NN upgrade to terminate the process after
upgrade on NN is completed. Contributed by Danilo Vunjak. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611723)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartupOptionUpgrade.java
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0, 2.5.0
>Reporter: Danilo Vunjak
>Assignee: Danilo Vunjak
> Fix For: 3.0.0, 2.6.0
>
> Attachments: HDFS-6597.04.patch, HDFS-6597.05.patch,
> HDFS-6597.06.patch, JIRA-HDFS-30.patch, JIRA-HDFS-6597.02.patch,
> JIRA-HDFS-6597.03.patch, JIRA-HDFS-6597.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066560#comment-14066560
]
Hudson commented on HDFS-6597:
--
FAILURE: Integrated in Hadoop-trunk-Commit #5912 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5912/])
HDFS-6597. Add a new option to NN upgrade to terminate the process after
upgrade on NN is completed. Contributed by Danilo Vunjak. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611723)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartupOptionUpgrade.java
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0, 2.5.0
>Reporter: Danilo Vunjak
>Assignee: Danilo Vunjak
> Fix For: 3.0.0, 2.6.0
>
> Attachments: HDFS-6597.04.patch, HDFS-6597.05.patch,
> HDFS-6597.06.patch, JIRA-HDFS-30.patch, JIRA-HDFS-6597.02.patch,
> JIRA-HDFS-6597.03.patch, JIRA-HDFS-6597.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066312#comment-14066312
]
Hadoop QA commented on HDFS-6597:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656492/HDFS-6597.06.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7384//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7384//console
This message is automatically generated.
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Danilo Vunjak
>Assignee: Danilo Vunjak
> Attachments: HDFS-6597.04.patch, HDFS-6597.05.patch,
> HDFS-6597.06.patch, JIRA-HDFS-30.patch, JIRA-HDFS-6597.02.patch,
> JIRA-HDFS-6597.03.patch, JIRA-HDFS-6597.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063889#comment-14063889
]
Hadoop QA commented on HDFS-6597:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12656062/HDFS-6597.05.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA
org.apache.hadoop.hdfs.server.namenode.TestProcessCorruptBlocks
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7362//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7362//console
This message is automatically generated.
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Danilo Vunjak
>Assignee: Danilo Vunjak
> Attachments: HDFS-6597.04.patch, HDFS-6597.05.patch,
> JIRA-HDFS-30.patch, JIRA-HDFS-6597.02.patch, JIRA-HDFS-6597.03.patch,
> JIRA-HDFS-6597.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063714#comment-14063714
]
Chris Nauroth commented on HDFS-6597:
-
Thanks, Danilo. This is looking good. Just 2 more comments:
# In {{TestStartupOptionUpgrade}}, please remove the unused import of
{{HttpConfig}}.
# Also in the test, the {{setUp}} method still sets {{startOpt}} to {{UPGRADE}}
before each test runs. I think you'll need to remove that line.
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Danilo Vunjak
>Assignee: Danilo Vunjak
> Attachments: HDFS-6597.04.patch, HDFS-6597.05.patch,
> JIRA-HDFS-30.patch, JIRA-HDFS-6597.02.patch, JIRA-HDFS-6597.03.patch,
> JIRA-HDFS-6597.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063658#comment-14063658
]
Danilo Vunjak commented on HDFS-6597:
-
Hi [~cnauroth],
I did changes as you suggest.
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Danilo Vunjak
>Assignee: Danilo Vunjak
> Attachments: HDFS-6597.04.patch, HDFS-6597.05.patch,
> JIRA-HDFS-30.patch, JIRA-HDFS-6597.02.patch, JIRA-HDFS-6597.03.patch,
> JIRA-HDFS-6597.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062687#comment-14062687
]
Chris Nauroth commented on HDFS-6597:
-
Hi, [~dvunjak]. This mostly looks good. I have 2 comments:
# I built the distro after adding a bogus new feature to bump the version to
-58 in {{NameNodeLayoutVersion}}. I tried running -upgradeOnly, but it didn't
actually upgrade the metadata files. It looks like you'll need another change
in {{FSImage#recoverTransitionRead}}. There is a switch statement that looks
for the {{UPGRADE}} option, but not the new {{UPGRADEONLY}} option.
# The new test suite is a copy of the existing {{TestStartupOptionUpgrade}}
with the option changed to {{UPGRADEONLY}}. Instead of cloning the code, this
looks like a good opportunity for a JUnit {{Parameterized}} test. See
{{TestNameNodeHttpServer}} for an existing example of a {{Parameterized}} test.
I think you can make a fairly small change in the existing
{{TestStartupOptionUpgrade}} so that it's parameterized to run on both options:
{{UPGRADE}} and {{UPGRADEONLY}}.
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Danilo Vunjak
>Assignee: Danilo Vunjak
> Attachments: HDFS-6597.04.patch, JIRA-HDFS-30.patch,
> JIRA-HDFS-6597.02.patch, JIRA-HDFS-6597.03.patch, JIRA-HDFS-6597.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062471#comment-14062471
]
Hadoop QA commented on HDFS-6597:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12655785/HDFS-6597.04.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7349//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7349//console
This message is automatically generated.
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Danilo Vunjak
> Attachments: HDFS-6597.04.patch, JIRA-HDFS-30.patch,
> JIRA-HDFS-6597.02.patch, JIRA-HDFS-6597.03.patch, JIRA-HDFS-6597.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062145#comment-14062145
]
Hadoop QA commented on HDFS-6597:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12655777/JIRA-HDFS-6597.02.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7348//console
This message is automatically generated.
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Danilo Vunjak
> Attachments: JIRA-HDFS-30.patch, JIRA-HDFS-6597.02.patch,
> JIRA-HDFS-6597.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055382#comment-14055382
]
Chris Nauroth commented on HDFS-6597:
-
The new patch proposal for naming the option "-upgradeOnly" looks good to me.
I'd say it would be good to go ahead with full implementation using that
naming. Thanks, [~dvunjak]!
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Danilo Vunjak
> Attachments: JIRA-HDFS-30.patch, JIRA-HDFS-6597.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14047623#comment-14047623
]
Danilo Vunjak commented on HDFS-6597:
-
Hi,
I've attached patch proposal. Please take a look.
Thanks,
Danilo
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Danilo Vunjak
> Attachments: JIRA-HDFS-30.patch, JIRA-HDFS-6597.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043654#comment-14043654
]
Danilo Vunjak commented on HDFS-6597:
-
Hi guys,
You have point when saying -force is not right name. I would pick -upgradeOnly
as maybe best option. What is your opinion?
[~jingzhao]
Yes, NN is included in all services. Datanodes need it up to upgrade itself.
Thanks,
Danilo
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Danilo Vunjak
> Attachments: JIRA-HDFS-30.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14042602#comment-14042602
]
Jing Zhao commented on HDFS-6597:
-
bq. and later when all services were started, upgrade of datanodes finished
sucessfully and system run
Here is NN still included in "all services"? If it's not, how will upgrade of
datanodes be handled?
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Danilo Vunjak
> Attachments: JIRA-HDFS-30.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6597) Add a new option to NN upgrade to terminate the process after upgrade on NN is completed
[
https://issues.apache.org/jira/browse/HDFS-6597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14042525#comment-14042525
]
Mit Desai commented on HDFS-6597:
-
Changing the summary to describe the jira more accurately.
> Add a new option to NN upgrade to terminate the process after upgrade on NN
> is completed
>
>
> Key: HDFS-6597
> URL: https://issues.apache.org/jira/browse/HDFS-6597
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Danilo Vunjak
> Attachments: JIRA-HDFS-30.patch
>
>
> Currently when namenode is started for upgrade (hadoop namenode -upgrade
> command), after finishing upgrade of metadata, namenode starts working
> normally and wait for datanodes to upgrade itself and connect to to NN. We
> need to have option for upgrading only NN metadata, so after upgrade is
> finished on NN, process should terminate.
> I have tested it by changing in file: hdfs.server.namenode.NameNode.java,
> method: public static NameNode createNameNode(String argv[], Configuration
> conf):
> in switch added
> case UPGRADE:
> case UPGRADE:
> {
> DefaultMetricsSystem.initialize("NameNode");
> NameNode nameNode = new NameNode(conf);
> if (startOpt.getForceUpgrade()) {
> terminate(0);
> return null;
> }
>
> return nameNode;
> }
> This did upgrade of metadata, closed process after finished, and later when
> all services were started, upgrade of datanodes finished sucessfully and
> system run .
> What I'm suggesting right now is to add new startup parameter "-force", so
> namenode can be started like this "hadoop namenode -upgrade -force", so we
> can indicate that we want to terminate process after upgrade metadata on NN
> is finished. Old functionality should be preserved, so users can run "hadoop
> namenode -upgrade" on same way and with same behaviour as it was previous.
> Thanks,
> Danilo
--
This message was sent by Atlassian JIRA
(v6.2#6252)
