[ https://issues.apache.org/jira/browse/YARN-9927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339383#comment-17339383 ]
Qi Zhu edited comment on YARN-9927 at 5/6/21, 6:41 AM: ------------------------------------------------------- Great review and investigation! Thanks very much [~ebadger] [~gandras] . I agree with you that we should do some stress test done via SLS or manually. And the more generic way of event handling is a great improvement in YARN. I will investigate how to use SLS to confirm the improvement. And about the test, i will change it to test both the multi-thread and the single one. was (Author: zhuqi): Great review and investigation! Thanks very much [~ebadger] [~ebadger] . I agree with you that we should do some stress test done via SLS or manually. And the more generic way of event handling is a great improvement in YARN. I will investigate how to use SLS to confirm the improvement. And about the test, i will change it to test both the multi-thread and the single one. > RM multi-thread event processing mechanism > ------------------------------------------ > > Key: YARN-9927 > URL: https://issues.apache.org/jira/browse/YARN-9927 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn > Affects Versions: 3.0.0, 2.9.2 > Reporter: hcarrot > Assignee: Qi Zhu > Priority: Major > Attachments: RM multi-thread event processing mechanism.pdf, > YARN-9927.001.patch, YARN-9927.002.patch, YARN-9927.003.patch, > YARN-9927.004.patch, YARN-9927.005.patch > > > Recently, we have observed serious event blocking in RM event dispatcher > queue. After analysis of RM event monitoring data and RM event processing > logic, we found that > 1) environment: a cluster with thousands of nodes > 2) RMNodeStatusEvent dominates 90% time consumption of RM event scheduler > 3) Meanwhile, RM event processing is in a single-thread mode, and It results > in the low headroom of RM event scheduler, thus performance of RM. > So we proposed a RM multi-thread event processing mechanism to improve RM > performance. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org