[ https://issues.apache.org/jira/browse/YARN-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168274#comment-15168274 ]
Jun Gong commented on YARN-4720: -------------------------------- Thanks [~mingma] for suggestions. I attached a wrong version patch(YARN-4720.04.patch)... The new patch fixed the problem and added a new test. > Skip unnecessary NN operations in log aggregation > ------------------------------------------------- > > Key: YARN-4720 > URL: https://issues.apache.org/jira/browse/YARN-4720 > Project: Hadoop YARN > Issue Type: Improvement > Reporter: Ming Ma > Assignee: Jun Gong > Attachments: YARN-4720.01.patch, YARN-4720.02.patch, > YARN-4720.03.patch, YARN-4720.04.patch, YARN-4720.05.patch > > > Log aggregation service could have unnecessary NN operations in the following > scenarios: > * No new local log has been created since the last upload for the long > running service scenario. > * NM uses {{ContainerLogAggregationPolicy}} that skips log aggregation for > certain containers. > In the following code snippet, even though {{pendingContainerInThisCycle}} is > empty, it still creates the writer and then removes the file later. Thus it > introduces unnecessary create/getfileinfo/delete NN calls when NM doesn't > aggregate logs for an app. > > {noformat} > AppLogAggregatorImpl.java > ...... > writer = > new LogWriter(this.conf, this.remoteNodeTmpLogFileForApp, > this.userUgi); > ...... > for (ContainerId container : pendingContainerInThisCycle) { > ...... > } > ...... > if (remoteFS.exists(remoteNodeTmpLogFileForApp)) { > if (rename) { > remoteFS.rename(remoteNodeTmpLogFileForApp, renamedPath); > } else { > remoteFS.delete(remoteNodeTmpLogFileForApp, false); > } > } > ...... > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)