[
https://issues.apache.org/jira/browse/YARN-8199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885150#comment-16885150
]
Adam Antal commented on YARN-8199:
----------------------------------
Thanks for patch [~Prabhu Joseph],
1) When one of the {{getFileStatus}} call fails all of the other files are not
getting processed since the catch block is out of the loop. That would not be a
problem for HDFS, but if another filesystem is given as remote-app-folder where
{{getFileStatus}} is not reliable then it can produce unexpected messages
(S3!). Also it would make this debugging not reliable at all: if a file is not
mentioned there in the logs, we could not deduce that this log has actually not
exceeded the limit given with the configuration.
2) I agree putting this to a debug block, since the {{getFileStatus}} reasoning
above. However I'd recommend mentioning this in the description: it only takes
effect if level at least debug (even though the configuration itself implicitly
contains "debug").
3) Could please add the container id (and maybe the application id as well)
when displaying the message about the size of the file? If one app has lots of
containers some of those having syslog exceeding the size then the error
message can not be bound to a container.
4) I'd rename yarn.log-aggregation.debug.filesize.bytes to
yarn.log-aggregation.debug.filesize since you already mention in the
description the measure.
Just out of curiosity: how did you end up with 100Mb as the limit?
> Logging fileSize of log files under NM Local Dir
> ------------------------------------------------
>
> Key: YARN-8199
> URL: https://issues.apache.org/jira/browse/YARN-8199
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: log-aggregation
> Affects Versions: 2.7.3
> Reporter: Prabhu Joseph
> Assignee: Prabhu Joseph
> Priority: Major
> Labels: supportability
> Attachments: 0001-YARN-8199.patch, 0002-YARN-8199.patch,
> YARN-8199-003.patch
>
>
> Logging fileSize of log files like syslog, stderr, stdout under NM Local Dir
> by NodeManager before the cleanup will help to find the application which has
> written too verbose.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
