[
https://issues.apache.org/jira/browse/YARN-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14326912#comment-14326912
]
Vinod Kumar Vavilapalli commented on YARN-2942:
-----------------------------------------------
Apologies for coming in real late. I've been thinking about this problem for a
long time, since before YARN came to Apache :)
I think HDFS-3689 will help a lot in this area. Offline I was requesting HDFS
folks to help make progress there. Now that that got in, I think we should
consider using that as the first step. It should help reduce the file-count
completely, even though the block count problem is still unresolved. The long
term solution for the later really is HDFS supporting atomic append (with
concurrent writers) - it's better to get the problem fixed at the storage layer.
We should try to avoid rereading the entire log file and rewriting again. How
about we try the concat approach (with variable length blocks) first before we
try the reread+rewrite?
> Aggregated Log Files should be combined
> ---------------------------------------
>
> Key: YARN-2942
> URL: https://issues.apache.org/jira/browse/YARN-2942
> Project: Hadoop YARN
> Issue Type: New Feature
> Affects Versions: 2.6.0
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Attachments: CombinedAggregatedLogsProposal_v3.pdf,
> CompactedAggregatedLogsProposal_v1.pdf,
> CompactedAggregatedLogsProposal_v2.pdf, YARN-2942-preliminary.001.patch,
> YARN-2942-preliminary.002.patch, YARN-2942.001.patch, YARN-2942.002.patch,
> YARN-2942.003.patch
>
>
> Turning on log aggregation allows users to easily store container logs in
> HDFS and subsequently view them in the YARN web UIs from a central place.
> Currently, there is a separate log file for each Node Manager. This can be a
> problem for HDFS if you have a cluster with many nodes as you’ll slowly start
> accumulating many (possibly small) files per YARN application. The current
> “solution” for this problem is to configure YARN (actually the JHS) to
> automatically delete these files after some amount of time.
> We should improve this by compacting the per-node aggregated log files into
> one log file per application.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
