[
https://issues.apache.org/jira/browse/YARN-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16659058#comment-16659058
]
Weiwei Yang commented on YARN-8925:
-----------------------------------
Hi [~Tao Yang]
Good catch! I think this was discussed sometime before. Cc [~Naganarasimha],
[~sunilg].
I just did a quick look at your patch, following comparing code
{code:java}
NodeAttributesUtils.isNodeAttributesEquals(nodeAttributes,
currentNodeAttributes))
{code}
in added in each {{nodeHeartbeat}} call.
Instead of doing that, can we move the logic to NM side. See similar code in
{{NodeStatusUpdaterImpl#getNodeLabelsForHeartbeat}}. I think this way it avoids
the overhead as the comparing in RM side is still expensive.
Does that make sense to you?
BTW, I am moving this under umbrella YARN-8766.
Thanks
> Updating distributed node attributes only when necessary
> --------------------------------------------------------
>
> Key: YARN-8925
> URL: https://issues.apache.org/jira/browse/YARN-8925
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
> Affects Versions: 3.2.1
> Reporter: Tao Yang
> Assignee: Tao Yang
> Priority: Major
> Attachments: YARN-8925.001.patch
>
>
> Currently if distributed node attributes exist, even though there is no
> change, updating for distributed node attributes will happen in every
> heartbeat between NM and RM. Updating process will hold
> NodeAttributesManagerImpl#writeLock and may have some influence in a large
> cluster. We have found nodes UI of a large cluster is opened slowly and most
> time it's waiting for the lock in NodeAttributesManagerImpl. I think this
> updating should be called only when necessary to enhance the performance of
> related process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
