[
https://issues.apache.org/jira/browse/MAPREDUCE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov updated MAPREDUCE-5672:
-------------------------------------
Attachment: MAPREDUCE-5672.v06.patch
[~jlowe], thanks for the feedback. I removed the dedicated conf for MR-AM from
this patch, and added some doc to mapred-default.xml.
The point I would make for a dedicated conf for MR-AM is that there are much
less AM containers than MR task attempt containers. You might be interested in
always being able to see the full job in progress log, but you would be fine
with keeping just tails of task attempt logs. In MRv1, the job log is in JT
and would not be affected by mapreduce.task.userlog.limit.kb. However, since
there is no dedicated limit.kb for AM (though it is certainly not userlog),
this might be worth another JIRA dealing with both settings.
> Provide optional RollingFileAppender for container log4j (syslog)
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-5672
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5672
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am, mrv2
> Affects Versions: 2.2.0
> Reporter: Gera Shegalov
> Assignee: Gera Shegalov
> Attachments: MAPREDUCE-5672.v01.patch, MAPREDUCE-5672.v02.patch,
> MAPREDUCE-5672.v03.patch, MAPREDUCE-5672.v04.patch, MAPREDUCE-5672.v05.patch,
> MAPREDUCE-5672.v06.patch, Screen Shot 2013-12-05 at 3.21.02 PM.png, Screen
> Shot 2013-12-05 at 3.23.33 PM.png
>
>
> This JIRA is an alternative take on YARN-1130
> We propose providing an option of using a RollingFileAppender(RFA)-based
> implementation of container log appender as means of log size control via
> mapreduce.task.userlog.limit.kb.
> The idea is to use mapreduce.task.userlog.limit.kb as maximumFileSize of RFA.
> In addition yarn.app.mapreduce.container.log.backups (task attempt
> containers) and yarn.app.mapreduce.am.log.backups (MR-AM) are passed as
> maxBackupIndex.
> Both current ContainerLogAppender (CLA) and new ContainerRollingLogAppender
> (CRLA) co-exist. CLA is the default. CRLA is chosen when
> mapreduce.task.userlog.limit.kb > 0 && *.backups > 0.
> Pros:
> 1) CRLA output is visible in UI right away. CLA output with
> mapreduce.task.userlog.limit.kb > 0 is not visible until the task attempt
> finishes that prevents timely diagnostics.
> 2) Even with excessive logging and a large mapreduce.task.userlog.limit.kb,
> no space is taken from the JVM heap.
> 3) No UI impact, since YARN is already designed to deal with any log name
> beyond stderr/out, syslog, debug.out, profile.out
> Cons:
> 1) if the logging is excessive there will be more local filesystem metadata
> I/O due to roll. That should be negligible in the grand scheme.
> Furthermore, to improve log consistency and completeness in the case of JVM
> crashes and SIGTERMing by NM, we propose to restore the MRv1 behavior of
> periodic log syncing (every 5s) and having log sync as part of a shutdown
> hook.
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
