[ https://issues.apache.org/jira/browse/MAPREDUCE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855746#action_12855746 ]
Vinod K V commented on MAPREDUCE-1648: -------------------------------------- OK, I cannot apply the patch since it isn't for apache svn trunk, so I still cannot comment whether your explanations validate my questions. Please supply a trunk patch atleast for demonstration at your own time. In the interim, please make sure that we have the complete picture of logging for java, streaming *and* pipe tasks with *and* without reuse of JVMs (HADOOP-249). Also as Koji rightly mentioned above, we need to performant - HADOOP-1553. > Use rolling to limit tasklogs > ----------------------------- > > Key: MAPREDUCE-1648 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1648 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker > Reporter: Guilin Sun > Priority: Minor > Attachments: syslog.patch > > > There are at least two types of task-logs: syslog and stdlog > Task-Jvm outputs syslog by log4j with TaskLogAppender, TaskLogAppender looks > just like "tail -c", it stores last N byte/line logs in memory(via queue), > and do real output only if all logs is commit and Appender is going to close. > The common problem of TaskLogAppender and 'tail -c' is keep everything in > memory and user can't see any log output while task is in progress. > So I'm going to try RollingFileAppender instead of TaskLogAppender, use > MaxFileSize&MaxBackupIndex to limit log file size. > RollingFileAppender is also suitable for stdout/stderr, just redirect > stdout/stderr to log4j via LoggingOutputStream, no client code have to be > changed, and RollingFileAppender seems better than 'tail -c' too. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira