[
https://issues.apache.org/jira/browse/YARN-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479880#comment-13479880
]
Siddharth Seth commented on YARN-171:
-------------------------------------
Ignore the last comment. At least the bit about the NM not serving logs after
App completion.
When aggregation is disabled, the NM is keeping track of applications till the
logs are active on the NM - which sets it up nicely for an OOM on busy
clusters, depending on the log-retention time.
Accessing the logs requires some application info - ACLs, owner. Maintaining
this in memory for the log-retention time could be problematic. One option
could be to create an index similar to the history server on local FS. Would
prefer not creating a single file per app though. Any suggestions on a
lightweight key value store with a compatible license ?
> NodeManager should serve logs directly if log-aggregation is not enabled
> ------------------------------------------------------------------------
>
> Key: YARN-171
> URL: https://issues.apache.org/jira/browse/YARN-171
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Siddharth Seth
>
> NodeManagers never serve logs for completed applications. If log-aggregation
> is not enabled, in the interim, due to bugs like YARN-162, this is a serious
> problem for users as logs are necessarily not available.
> We should let nodes serve logs directly if
> YarnConfiguration.LOG_AGGREGATION_ENABLED is set. This should be okay as
> NonAggregatingLogHandler can retain logs upto
> YarnConfiguration.NM_LOG_RETAIN_SECONDS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
