Our standby namenode crashed and we couldn't restart the standby namenode. I tried to manually restart it but still it is failed. How do I recover and restart the standby namenode.
2016-05-20 18:53:57,954 INFO namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream ' http://usw2stdpma01.glassdoor.local:8480/getJournal?jid=dfs-nameservices&segmentTxId=14726901&storageInfo=-60%3A761966699%3A0%3ACID-d16e0895-7c12-404e-9223-952d1b19ace0' to transaction ID 13013207 2016-05-20 18:53:58,216 WARN namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(750)) - Encountered exception loading fsimage java.io.IOException: There appears to be a gap in the edit log. We expected txid 13013207, but got txid 14726901. at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:212) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:829) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:684) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1032) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:748) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:538) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:597) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:764) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:748) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1441) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1507) 2016-05-20 18:53:58,322 FATAL namenode.NameNode (NameNode.java:main(1512)) - Failed to start namenode. java.io.IOException: There appears to be a gap in the edit log. We expected txid 13013207, but got txid 14726901. at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:212) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:829) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:684) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1032) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:748) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:538) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:597) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:764) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:748) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1441) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1507) 2016-05-20 18:53:58,324 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 2016-05-20 18:53:58,325 INFO namenode.NameNode (StringUtils.java:run(659)) - SHUTDOWN_MSG:
