[
https://issues.apache.org/jira/browse/HADOOP-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888105#action_12888105
]
Andrew Ryan commented on HADOOP-6858:
-------------------------------------
We chose not to go that route, because date stamping in that way would address
the "logs get blown away on startup" problem, but then it creates a "gc logs
are never cleaned up automatically by hadoop" problem. But there are certainly
many ways to solve this problem. We're hoping to reach some kind of consensus.
> Enable rotateable JVM garbage collection logs for Hadoop daemons
> ----------------------------------------------------------------
>
> Key: HADOOP-6858
> URL: https://issues.apache.org/jira/browse/HADOOP-6858
> Project: Hadoop Common
> Issue Type: New Feature
> Components: scripts
> Affects Versions: 0.22.0
> Reporter: Andrew Ryan
> Attachments: HADOOP-6858.patch
>
>
> The purpose of this enhancement is to make it easier to collect garbage
> collection logs and insure that they persist across restarts in the same way
> that the standard output files of Hadoop daemon JVM's currently does.
> Garbage collection logs are a vital debugging tool for administrators and
> developers. In our production environments, at some point or another, every
> single type of Hadoop daemon has OOM'ed or experienced other significant
> issues related to GC and/or lack of heap memory. For the longest time, we
> have put in garbage collection logs in our HADOOP_NAMENODE_OPTS,
> HADOOP_JOBTRACKER_OPTS, etc. by using options like "-XX:+PrintGCDateStamps
> -XX:+PrintGCDetails -Xloggc:$HADOOP_LOG_DIR/jobtracker.gc.log".
> Unfortunately, these logs don't survive a restart of the node, so if a node
> OOM's and then is restarted automatically, or manually by someone who is
> unaware, we lose the GC logs forever. We also have to manually add GC log
> options to each daemon. This patch:
> 1) Creates a single, optional, off by default, parameter for specifying GC
> logging.
> 2) If that parameter is set, automatically enables GC logging for all daemons
> in the cluster. The parameter is flexible enough to allow for the different
> ways various vendor's JVM's require garbage collection logging to be
> specified.
> 3) If GC logging is on, insures that the GC log files for each daemon are
> rotated with up to 5 copies kept, same as the .out files currently.
> We are currently running a variation of this patch in our 0.20 install. This
> patch actually includes changes to common, mapred, and hdfs, so it obviously
> cannot be applied as-is, but is included here for review and comments.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
