[
https://issues.apache.org/jira/browse/YARN-9927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962135#comment-16962135
]
Eric Payne commented on YARN-9927:
----------------------------------
Thank you [~hcarrot] for bringing this up and for your work in making the RM
better, and thank you for providing the architecture document.
{quote}
bq. we just replace the time in the dispatcher queue with lock-holding time for
each event.
RM can process different events concurrently
{quote}
I share the same concern as [~adam.antal]. Since the code path through
RMNodeStatusEvent is protected by locks, I think that even if multiple
RMNodeStatusEvent events are being processed by multiple dispatcher threads at
the same time, only one will actually be running. Unless the design is to
dedicate one thread to handling only RMNodeStatusEvent events and the other
threads to handling non-RMNodeStatusEvent events.
I look forward to seeing your POC.
> RM multi-thread event processing mechanism
> ------------------------------------------
>
> Key: YARN-9927
> URL: https://issues.apache.org/jira/browse/YARN-9927
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
> Affects Versions: 3.0.0, 2.9.2
> Reporter: hcarrot
> Priority: Major
> Attachments: RM multi-thread event processing mechanism.pdf
>
>
> Recently, we have observed serious event blocking in RM event dispatcher
> queue. After analysis of RM event monitoring data and RM event processing
> logic, we found that
> 1) environment: a cluster with thousands of nodes
> 2) RMNodeStatusEvent dominates 90% time consumption of RM event scheduler
> 3) Meanwhile, RM event processing is in a single-thread mode, and It results
> in the low headroom of RM event scheduler, thus performance of RM.
> So we proposed a RM multi-thread event processing mechanism to improve RM
> performance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
