Re: HDFS upgrade problem of fsImage
Yes realized that and I see your point :-) However seems like some fs
inconsistency present, did you attempt rollback/finalizeUpgrade and check?
For that error, FSImage.java/code finds a previous fs state -
// Upgrade is allowed only if there are
// no previous fs states in any of the directories
for (IteratorStorageDirectory it = storage.dirIterator(); it.hasNext();) {
StorageDirectory sd = it.next();
if (sd.getPreviousDir().exists())
throw new InconsistentFSStateException(sd.getRoot(),
previous fs state should not exist during upgrade.
+ Finalize or rollback first.);
}
Thanks
Rekha
From: Azuryy Yu azury...@gmail.commailto:azury...@gmail.com
Reply-To: user@hadoop.apache.orgmailto:user@hadoop.apache.org
user@hadoop.apache.orgmailto:user@hadoop.apache.org
Date: Thursday 21 November 2013 5:19 PM
To: user@hadoop.apache.orgmailto:user@hadoop.apache.org
user@hadoop.apache.orgmailto:user@hadoop.apache.org
Cc: hdfs-...@hadoop.apache.orgmailto:hdfs-...@hadoop.apache.org
hdfs-...@hadoop.apache.orgmailto:hdfs-...@hadoop.apache.org
Subject: Re: HDFS upgrade problem of fsImage
I insist hot upgrade on the test cluster because I want hot upgrade on the prod
cluster.
On 2013-11-21 7:23 PM, Joshi, Rekha
rekha_jo...@intuit.commailto:rekha_jo...@intuit.com wrote:
Hi Azurry,
This error occurs when FSImage finds previous fs state, and as log states you
would need to either finalizeUpgrade or rollback to proceed.Below -
bin/hadoop dfsadmin –finalizeUpgrade
hadoop dfsadmin –rollback
On side note for a small test cluster on which one might suspect you are the
only user, why wouldn't you insist on hot upgrade? :-)
Thanks
Rekha
Some helpful guidelines for upgrade here -
http://wiki.apache.org/hadoop/Hadoop_Upgrade
https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
From: Azuryy Yu azury...@gmail.commailto:azury...@gmail.com
Reply-To: user@hadoop.apache.orgmailto:user@hadoop.apache.org
user@hadoop.apache.orgmailto:user@hadoop.apache.org
Date: Thursday 21 November 2013 9:48 AM
To: hdfs-...@hadoop.apache.orgmailto:hdfs-...@hadoop.apache.org
hdfs-...@hadoop.apache.orgmailto:hdfs-...@hadoop.apache.org,
user@hadoop.apache.orgmailto:user@hadoop.apache.org
user@hadoop.apache.orgmailto:user@hadoop.apache.org
Subject: HDFS upgrade problem of fsImage
Hi Dear,
I have a small test cluster with hadoop-2.0x, and HA configuraded, but I want
to upgrade to hadoop-2.2.
I dont't want to stop cluster during upgrade, so my steps are:
1) on standby NN: hadoop-dameon.sh stop namenode
2) remove HA configuration in the conf
3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
but Exception in the NN log, so how to upgrade and don't stop the whole cluster.
Thanks.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory
/hdfs/name is in an inconsistent state: previous fs state should not exist
during upgrade. Finalize or rollback first.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:692)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:677)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
Re: HDFS upgrade problem of fsImage
Thanks Joshi,
Maybe I pasted wrong log messages.
please looked at here for the real story.
https://issues.apache.org/jira/browse/HDFS-5550
On Fri, Nov 22, 2013 at 6:25 PM, Joshi, Rekha rekha_jo...@intuit.comwrote:
Yes realized that and I see your point :-) However seems like some fs
inconsistency present, did you attempt rollback/finalizeUpgrade and check?
For that error, FSImage.java/code finds a previous fs state -
// Upgrade is allowed only if there are
// no previous fs states in any of the directories
for (IteratorStorageDirectory it = storage.dirIterator();
it.hasNext();) {
StorageDirectory sd = it.next();
if (sd.getPreviousDir().exists())
throw new InconsistentFSStateException(sd.getRoot(),
previous fs state should not exist during upgrade.
+ Finalize or rollback first.);
}
Thanks
Rekha
From: Azuryy Yu azury...@gmail.com
Reply-To: user@hadoop.apache.org user@hadoop.apache.org
Date: Thursday 21 November 2013 5:19 PM
To: user@hadoop.apache.org user@hadoop.apache.org
Cc: hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org
Subject: Re: HDFS upgrade problem of fsImage
I insist hot upgrade on the test cluster because I want hot upgrade on
the prod cluster.
On 2013-11-21 7:23 PM, Joshi, Rekha rekha_jo...@intuit.com wrote:
Hi Azurry,
This error occurs when FSImage finds previous fs state, and as log states
you would need to either finalizeUpgrade or rollback to proceed.Below -
bin/hadoop dfsadmin –finalizeUpgrade
hadoop dfsadmin –rollback
On side note for a small test cluster on which one might suspect you are the
only user, why wouldn't you insist on hot upgrade? :-)
Thanks
Rekha
Some helpful guidelines for upgrade here -
http://wiki.apache.org/hadoop/Hadoop_Upgrade
https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation
From: Azuryy Yu azury...@gmail.com
Reply-To: user@hadoop.apache.org user@hadoop.apache.org
Date: Thursday 21 November 2013 9:48 AM
To: hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org,
user@hadoop.apache.org user@hadoop.apache.org
Subject: HDFS upgrade problem of fsImage
Hi Dear,
I have a small test cluster with hadoop-2.0x, and HA configuraded, but
I want to upgrade to hadoop-2.2.
I dont't want to stop cluster during upgrade, so my steps are:
1) on standby NN: hadoop-dameon.sh stop namenode
2) remove HA configuration in the conf
3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster
but Exception in the NN log, so how to upgrade and don't stop the whole
cluster.
Thanks.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
Directory /hdfs/name is in an inconsistent state: previous fs state should
not exist during upgrade. Finalize or rollback first.
at
org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:692)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:677)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
Re: HDFS upgrade problem of fsImage
Hi Azurry, This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below - bin/hadoop dfsadmin –finalizeUpgrade hadoop dfsadmin –rollback On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-) Thanks Rekha Some helpful guidelines for upgrade here - http://wiki.apache.org/hadoop/Hadoop_Upgrade https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation From: Azuryy Yu azury...@gmail.commailto:azury...@gmail.com Reply-To: user@hadoop.apache.orgmailto:user@hadoop.apache.org user@hadoop.apache.orgmailto:user@hadoop.apache.org Date: Thursday 21 November 2013 9:48 AM To: hdfs-...@hadoop.apache.orgmailto:hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.orgmailto:hdfs-...@hadoop.apache.org, user@hadoop.apache.orgmailto:user@hadoop.apache.org user@hadoop.apache.orgmailto:user@hadoop.apache.org Subject: HDFS upgrade problem of fsImage Hi Dear, I have a small test cluster with hadoop-2.0x, and HA configuraded, but I want to upgrade to hadoop-2.2. I dont't want to stop cluster during upgrade, so my steps are: 1) on standby NN: hadoop-dameon.sh stop namenode 2) remove HA configuration in the conf 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster but Exception in the NN log, so how to upgrade and don't stop the whole cluster. Thanks. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hdfs/name is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first. at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:692) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:677) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
Re: HDFS upgrade problem of fsImage
Thanks Joshi, I don't have upgrade before. the test cluser is the new cluster with hadoop-2.0.3. so I don't need to 'bin/hadoop dfsadmin –finalizeUpgrade' On Thu, Nov 21, 2013 at 7:22 PM, Joshi, Rekha rekha_jo...@intuit.comwrote: Hi Azurry, This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below - bin/hadoop dfsadmin –finalizeUpgrade hadoop dfsadmin –rollback On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-) Thanks Rekha Some helpful guidelines for upgrade here - http://wiki.apache.org/hadoop/Hadoop_Upgrade https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation From: Azuryy Yu azury...@gmail.com Reply-To: user@hadoop.apache.org user@hadoop.apache.org Date: Thursday 21 November 2013 9:48 AM To: hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org, user@hadoop.apache.org user@hadoop.apache.org Subject: HDFS upgrade problem of fsImage Hi Dear, I have a small test cluster with hadoop-2.0x, and HA configuraded, but I want to upgrade to hadoop-2.2. I dont't want to stop cluster during upgrade, so my steps are: 1) on standby NN: hadoop-dameon.sh stop namenode 2) remove HA configuration in the conf 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster but Exception in the NN log, so how to upgrade and don't stop the whole cluster. Thanks. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hdfs/name is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first. at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:692) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:677) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
Re: HDFS upgrade problem of fsImage
I insist hot upgrade on the test cluster because I want hot upgrade on the prod cluster. On 2013-11-21 7:23 PM, Joshi, Rekha rekha_jo...@intuit.com wrote: Hi Azurry, This error occurs when FSImage finds previous fs state, and as log states you would need to either finalizeUpgrade or rollback to proceed.Below - bin/hadoop dfsadmin –finalizeUpgrade hadoop dfsadmin –rollback On side note for a small test cluster on which one might suspect you are the only user, why wouldn't you insist on hot upgrade? :-) Thanks Rekha Some helpful guidelines for upgrade here - http://wiki.apache.org/hadoop/Hadoop_Upgrade https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation From: Azuryy Yu azury...@gmail.com Reply-To: user@hadoop.apache.org user@hadoop.apache.org Date: Thursday 21 November 2013 9:48 AM To: hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org, user@hadoop.apache.org user@hadoop.apache.org Subject: HDFS upgrade problem of fsImage Hi Dear, I have a small test cluster with hadoop-2.0x, and HA configuraded, but I want to upgrade to hadoop-2.2. I dont't want to stop cluster during upgrade, so my steps are: 1) on standby NN: hadoop-dameon.sh stop namenode 2) remove HA configuration in the conf 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster but Exception in the NN log, so how to upgrade and don't stop the whole cluster. Thanks. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hdfs/name is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first. at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:323) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:248) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:858) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:620) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:445) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:494) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:692) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:677) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1345)
