[
https://issues.apache.org/jira/browse/YARN-9927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335804#comment-17335804
]
Eric Badger commented on YARN-9927:
-----------------------------------
{noformat}
+ // Test multi thread dispatcher
+ conf.setBoolean(YarnConfiguration.
+ MULTI_THREAD_DISPATCHER_ENABLED, true);
{noformat}
If this is a feature that is disabled by default, I don't think we should have
it enabled by default in all of the RM tests. I would be happier running it as
a parameterized test with both multi and single thread dispatchers.
In general I think the patch looks reasonable, but I would like to see testing
done to see if this makes the problem better or worse. I would think it would
make things better, but until we run some real tests on it, we won't really
know. So getting something similar to what [~hcarrot] provided originally would
be good. That way we can merge this with confidence.
> RM multi-thread event processing mechanism
> ------------------------------------------
>
> Key: YARN-9927
> URL: https://issues.apache.org/jira/browse/YARN-9927
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
> Affects Versions: 3.0.0, 2.9.2
> Reporter: hcarrot
> Assignee: Qi Zhu
> Priority: Major
> Attachments: RM multi-thread event processing mechanism.pdf,
> YARN-9927.001.patch, YARN-9927.002.patch, YARN-9927.003.patch,
> YARN-9927.004.patch, YARN-9927.005.patch
>
>
> Recently, we have observed serious event blocking in RM event dispatcher
> queue. After analysis of RM event monitoring data and RM event processing
> logic, we found that
> 1) environment: a cluster with thousands of nodes
> 2) RMNodeStatusEvent dominates 90% time consumption of RM event scheduler
> 3) Meanwhile, RM event processing is in a single-thread mode, and It results
> in the low headroom of RM event scheduler, thus performance of RM.
> So we proposed a RM multi-thread event processing mechanism to improve RM
> performance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
