[
https://issues.apache.org/jira/browse/YARN-2314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180575#comment-14180575
]
Jian He commented on YARN-2314:
-------------------------------
Thanks Jason, I looked at the patch, looks good overall. just one thing:
- IIUC, {{mayBeCloseProxy}} can be invoked by MR/NMClient, but
{{proxy.scheduledForClose}} is always false. So it won’t call the following
stopProxy. If cache is disabled, this doesn’t matter too much as the
idleTimeout is set to 0. But if the cache is enabled, MR/NMClient, won’t be
able to explicitly close the proxy ?
Also, Can you help me understand one point:
bq. See ClientCache.stopClient for details. Given that the whole point of the
ContainerManagementProtocolProxy cache is to preserve at least one reference to
the Client, the IPC Client stop method will never be called in practice and IPC
client threads will never be explicitly torn down as a result of calling
stopProxy.
once {{ContainerManagementProtocolProxy#tryCloseProxy}} is called, internally
it’ll call {{rpc.stopProxy}}, will it eventually call
{{ClientCache#stopClient}} ?
> ContainerManagementProtocolProxy can create thousands of threads for a large
> cluster
> ------------------------------------------------------------------------------------
>
> Key: YARN-2314
> URL: https://issues.apache.org/jira/browse/YARN-2314
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
> Affects Versions: 2.1.0-beta
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
> Attachments: YARN-2314.patch, YARN-2314v2.patch,
> disable-cm-proxy-cache.patch, nmproxycachefix.prototype.patch,
> tez-yarn-2314.xlsx
>
>
> ContainerManagementProtocolProxy has a cache of NM proxies, and the size of
> this cache is configurable. However the cache can grow far beyond the
> configured size when running on a large cluster and blow AM address/container
> limits. More details in the first comment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
