[ https://issues.apache.org/jira/browse/HDFS-7991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14556546#comment-14556546 ]
Allen Wittenauer commented on HDFS-7991: ---------------------------------------- bq. Just in your previous comment it seemed to me you did not even understand the issue . ... bq. 1. editlog had an issue and could not be consumed by 2NN or standby bq. 2. checkpointing is lagging behind (see HDFS-7609) bq. 3. There could many others bugs and issues (standby down etc) that could result in delayed checkpoint I've seen every single one of these in production, either at Y! or at LI during my time with Hadoop. My favorite is the time the regex was bugged so bad it caused the 2NN to crash during the log parsing because someone wrote a weird file name. So yeah, I'm pretty sure I do have a good grasp of exactly the issues you are talking about, having been on the receiving end of corrupted image files in the past and having to walk down to developer row to get them fixed. bq. No one is proposing that operators need do [safemode->checkpoint->shutdown] on every NN bring down. Oh? You mean like this completely broken code that is already sitting in trunk during the first attempt (HDFS-6353 ) to "fix" this issue? {code} if [[ "${COMMAND}" == "namenode" ]] && [[ "${HADOOP_DAEMON_MODE}" == "stop" ]]; then hadoop_debug "Do checkpoint if necessary before stopping NameNode" export CLASSPATH "${JAVA}" "-Dproc_dfsadmin" ${HADOOP_OPTS} "org.apache.hadoop.hdfs.tools.DFSAdmin" "-safemode" "enter" "${JAVA}" "-Dproc_dfsadmin" ${HADOOP_OPTS} "org.apache.hadoop.hdfs.tools.DFSAdmin" "-saveNamespace" "-beforeShutdown" "${JAVA}" "-Dproc_dfsadmin" ${HADOOP_OPTS} "org.apache.hadoop.hdfs.tools.DFSAdmin" "-safemode" "leave" fi {code} I'm glad that we agree that this code should get removed since it's causing so many problems. bq. In some cases checkpoint could not even be done because editlog was corrupt and could not be consumed by 2NN or standby (sorry, repeating myself). Yup, and in those cases, that's what they pay vendors to fix. For those of that don't, they roll back to the last good copy and move on. bq. This jira proposes to save namespace when checkpointing has not happened for a long time. All of the discussion up until recently has been about fixing the broken bits in the shell code. If we want to switch the discussion to make the namenode checkpoint optional when it's sent a kill, that's great. It means we can clean out the shell code and make this entirely a Java-level fix, as it should be. > Allow users to skip checkpoint when stopping NameNode > ----------------------------------------------------- > > Key: HDFS-7991 > URL: https://issues.apache.org/jira/browse/HDFS-7991 > Project: Hadoop HDFS > Issue Type: Improvement > Affects Versions: 3.0.0 > Reporter: Jing Zhao > Assignee: Jing Zhao > Attachments: HDFS-7991-shellpart.patch, HDFS-7991.000.patch, > HDFS-7991.001.patch, HDFS-7991.002.patch, HDFS-7991.003.patch, > HDFS-7991.004.patch > > > This is a follow-up jira of HDFS-6353. HDFS-6353 adds the functionality to > check if saving namespace is necessary before stopping namenode. As [~kihwal] > pointed out in this > [comment|https://issues.apache.org/jira/browse/HDFS-6353?focusedCommentId=14380898&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14380898], > in a secured cluster this new functionality requires the user to be kinit'ed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)