[ https://issues.apache.org/jira/browse/HDFS-11112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677124#comment-15677124 ]
Wei-Chiu Chuang commented on HDFS-11112: ---------------------------------------- Hi [~arpitagarwal] a while back I also studied the behavior of JN in this situation (with minor deviation) after finished HDFS-10360 to deal with the similar issue in DataNodes. Here's what I saw. Could you comment if this is also what you saw? Maybe the situation is different, and from what I saw JN didn't automatically format the directory. {quote} if current/VERSION is missing, it will start with no problem like nothing has happened. No auto-format. But when NN tries to send edits to the JN, the JN will respond with JournalNotFormattedException. Cloudera Manager dashboard will see JN and NN in bad health. 2016-06-07 11:14:18,656 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 8485, call org.apache.hadoop.hdfs.qjournal.protocol.QJournal Protocol.getJournalState from 172.26.10.80:36663 Call#7 Retry#0 org.apache.hadoop.hdfs.qjournal.protocol.JournalNotFormattedException: Journal Storage Directory /data/jn/nameservice1 not formatted at org.apache.hadoop.hdfs.qjournal.server.Journal.checkFormatted(Journal.java:472) at org.apache.hadoop.hdfs.qjournal.server.Journal.getLastPromisedEpoch(Journal.java:245) at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.getJournalState(JournalNodeRpcServer.java:124) at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.getJournalState(QJournalProtocolServerSideTranslat orPB.java:118) at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos. java:25415) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080) If I remove current/VERSION from all three JNs, NN will shutdown because it's unabel to get a quorum. If I put the file back to an active JN, it will still complain about it. But if I restart the JN with the file in place, the JN will function normally. So in the case of JN, there is no permanent harm and the damage can be reverted. {quote} [~linyiqun] I think the patch looks good to me overall. Do you know the behavior of JN after your patch? Does JN simply ignores format? Does JN continues to function normally? > Journal Nodes should refuse to format non-empty directories > ----------------------------------------------------------- > > Key: HDFS-11112 > URL: https://issues.apache.org/jira/browse/HDFS-11112 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: Arpit Agarwal > Assignee: Yiqun Lin > Attachments: HDFS-11112.001.patch, HDFS-11112.002.patch > > > Journal Nodes should reject the {{format}} RPC request if a storage directory > is non-empty. The relevant code is in {{JNStorage#format}}. > {code} > void format(NamespaceInfo nsInfo) throws IOException { > setStorageInfo(nsInfo); > ... > unlockAll(); > sd.clearDirectory(); > writeProperties(sd); > createPaxosDir(); > analyzeStorage(); > {code} > This would make the behavior similar to {{namenode -format -nonInteractive}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org