[
https://issues.apache.org/jira/browse/YARN-9927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17305895#comment-17305895
]
Qi Zhu edited comment on YARN-9927 at 3/22/21, 6:11 AM:
--------------------------------------------------------
Thanks [~gandras] for investigation and reply.
I agree with you that which you suggested is a better mode, the only concern to
me is that if the thread number will be too many because it will be consistent
with the number of EventType, and if it will cause some side effect compare
with the original mode.
I make sense to me after investigation, the number seems not a problem, we can
only add multi thread to those eventType which are in a big pressure.
But we also should make stress test to the new mode.
Let's wait for [~pbacsko] advice. :D
Thanks.
was (Author: zhuqi):
Thanks [~gandras] for investigation and reply.
I agree with you that which you suggested is a better mode, the only concern to
me is that if the thread number will be too many because it will be consistent
with the number of EventType, and if it will cause some side effect compare
with the original mode.
I make sense to me after investigation, the number seems not a problem, we can
only add multi thread to those eventType which are in a big pressure.
But we also should make stress test to the new mode.
Thanks.
> RM multi-thread event processing mechanism
> ------------------------------------------
>
> Key: YARN-9927
> URL: https://issues.apache.org/jira/browse/YARN-9927
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
> Affects Versions: 3.0.0, 2.9.2
> Reporter: hcarrot
> Assignee: Qi Zhu
> Priority: Major
> Attachments: RM multi-thread event processing mechanism.pdf,
> YARN-9927.001.patch
>
>
> Recently, we have observed serious event blocking in RM event dispatcher
> queue. After analysis of RM event monitoring data and RM event processing
> logic, we found that
> 1) environment: a cluster with thousands of nodes
> 2) RMNodeStatusEvent dominates 90% time consumption of RM event scheduler
> 3) Meanwhile, RM event processing is in a single-thread mode, and It results
> in the low headroom of RM event scheduler, thus performance of RM.
> So we proposed a RM multi-thread event processing mechanism to improve RM
> performance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
