[
https://issues.apache.org/jira/browse/HDFS-7991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14556546#comment-14556546
]
Allen Wittenauer commented on HDFS-7991:
----------------------------------------
bq. Just in your previous comment it seemed to me you did not even understand
the issue .
...
bq. 1. editlog had an issue and could not be consumed by 2NN or standby
bq. 2. checkpointing is lagging behind (see HDFS-7609)
bq. 3. There could many others bugs and issues (standby down etc) that could
result in delayed checkpoint
I've seen every single one of these in production, either at Y! or at LI during
my time with Hadoop. My favorite is the time the regex was bugged so bad it
caused the 2NN to crash during the log parsing because someone wrote a weird
file name.
So yeah, I'm pretty sure I do have a good grasp of exactly the issues you are
talking about, having been on the receiving end of corrupted image files in the
past and having to walk down to developer row to get them fixed.
bq. No one is proposing that operators need do
[safemode->checkpoint->shutdown] on every NN bring down.
Oh? You mean like this completely broken code that is already sitting in trunk
during the first attempt (HDFS-6353 ) to "fix" this issue?
{code}
if [[ "${COMMAND}" == "namenode" ]] &&
[[ "${HADOOP_DAEMON_MODE}" == "stop" ]]; then
hadoop_debug "Do checkpoint if necessary before stopping NameNode"
export CLASSPATH
"${JAVA}" "-Dproc_dfsadmin" ${HADOOP_OPTS}
"org.apache.hadoop.hdfs.tools.DFSAdmin" "-safemode" "enter"
"${JAVA}" "-Dproc_dfsadmin" ${HADOOP_OPTS}
"org.apache.hadoop.hdfs.tools.DFSAdmin" "-saveNamespace" "-beforeShutdown"
"${JAVA}" "-Dproc_dfsadmin" ${HADOOP_OPTS}
"org.apache.hadoop.hdfs.tools.DFSAdmin" "-safemode" "leave"
fi
{code}
I'm glad that we agree that this code should get removed since it's causing so
many problems.
bq. In some cases checkpoint could not even be done because editlog was corrupt
and could not be consumed by 2NN or standby (sorry, repeating myself).
Yup, and in those cases, that's what they pay vendors to fix. For those of
that don't, they roll back to the last good copy and move on.
bq. This jira proposes to save namespace when checkpointing has not happened
for a long time.
All of the discussion up until recently has been about fixing the broken bits
in the shell code. If we want to switch the discussion to make the namenode
checkpoint optional when it's sent a kill, that's great. It means we can clean
out the shell code and make this entirely a Java-level fix, as it should be.
> Allow users to skip checkpoint when stopping NameNode
> -----------------------------------------------------
>
> Key: HDFS-7991
> URL: https://issues.apache.org/jira/browse/HDFS-7991
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Jing Zhao
> Assignee: Jing Zhao
> Attachments: HDFS-7991-shellpart.patch, HDFS-7991.000.patch,
> HDFS-7991.001.patch, HDFS-7991.002.patch, HDFS-7991.003.patch,
> HDFS-7991.004.patch
>
>
> This is a follow-up jira of HDFS-6353. HDFS-6353 adds the functionality to
> check if saving namespace is necessary before stopping namenode. As [~kihwal]
> pointed out in this
> [comment|https://issues.apache.org/jira/browse/HDFS-6353?focusedCommentId=14380898&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14380898],
> in a secured cluster this new functionality requires the user to be kinit'ed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
