[
https://issues.apache.org/jira/browse/YARN-6277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950316#comment-15950316
]
Haibo Chen commented on YARN-6277:
----------------------------------
Thanks [~Feng Yuan] for reporting the issue and working on a patch! If I follow
you correctly, the AllocatorPerContext instance for NM_LOCAL_DIR is global, and
because ShuffleHandler and NM are not sharing the same configuration object,
ShuffleHandler does not see the change if the local directory is changed in NM,
resulting in ShuffleHandler and NM thinking of different values for NM_LOCAL_DIR
{code}
private Context confChanged(Configuration conf) throws IOException {
........
if (!newLocalDirs.equals(ctx.savedLocalDirs)) {
ctx = new Context();
String[] dirStrings = StringUtils.getTrimmedStrings(newLocalDirs);
ctx.localFS = FileSystem.getLocal(conf);
....
ctx.savedLocalDirs = newLocalDirs;
}
}
...
{code}
The if statement will always evaluate to true if the other has executed
confChanged() previously, so we have this thrashing issue? But looking at
FileSystem.getLocal() implementation though, it seems like that only when
caching for local file system is disabled, will massive number of
LocalFileSystem instances be created. Can you confirm is that your setting?
> Nodemanager heap memory leak
> ----------------------------
>
> Key: YARN-6277
> URL: https://issues.apache.org/jira/browse/YARN-6277
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.7.3, 2.8.1, 3.0.0-alpha2
> Reporter: Feng Yuan
> Assignee: Feng Yuan
> Attachments: YARN-6277.branch-2.8.001.patch
>
>
> Because LocalDirHandlerService@LocalDirAllocator`s mechanism,they will create
> massive LocalFileSystem.So lead to heap leak.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
