[
https://issues.apache.org/jira/browse/YARN-9927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hcarrot updated YARN-9927:
--------------------------
Description:
Recently, we have observed serious event blocking in RM event dispatcher queue.
After analysis of RM event monitoring data and RM event processing logic, we
found that
1) environment: a cluster with thousands of nodes
2) RMNodeStatusEvent dominates 90% time consumption of RM event scheduler
3) Meanwhile, RM event processing is in a single-thread mode, and It results in
the low headroom of RM event scheduler, thus performance of RM.
So we proposed a RM multi-thread event processing mechanism to improve RM
performance.
was:Recently, we have observed serious event blocking in RM event dispatcher
queue. After analysis of RM event monitoring data and RM event processing
logic, we found that the proportion of RMNodeStatusEvent is less than other
events, but the overall processing time of it is more than other events.
Meanwhile, RM event processing is in a single-thread mode, and It results in
the decrease of RM's performance. So we proposed a RM multi-thread event
processing mechanism to improve RM performance.
> RM multi-thread event processing mechanism
> ------------------------------------------
>
> Key: YARN-9927
> URL: https://issues.apache.org/jira/browse/YARN-9927
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
> Affects Versions: 3.0.0, 2.9.2
> Reporter: hcarrot
> Priority: Minor
> Attachments: RM multi-thread event processing mechanism.pdf
>
>
> Recently, we have observed serious event blocking in RM event dispatcher
> queue. After analysis of RM event monitoring data and RM event processing
> logic, we found that
> 1) environment: a cluster with thousands of nodes
> 2) RMNodeStatusEvent dominates 90% time consumption of RM event scheduler
> 3) Meanwhile, RM event processing is in a single-thread mode, and It results
> in the low headroom of RM event scheduler, thus performance of RM.
> So we proposed a RM multi-thread event processing mechanism to improve RM
> performance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
