[ https://issues.apache.org/jira/browse/YARN-9927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314294#comment-17314294 ]
Qi Zhu edited comment on YARN-9927 at 4/3/21, 4:24 PM: ------------------------------------------------------- [~gandras] [~pbacsko] Updated a patch, each eventType will have single async dispatcher thread. I will add unit test later. I think we first realize the multi thread dispatcher for each eventType, so that the event queue will not be a problem. And about the thread number, i counted the eventType number, it is in reasonable range. Then we can move to that, how can we consume the event queue faster in another jira. What's your opinion? :D Thanks. was (Author: zhuqi): [~gandras] [~pbacsko] Updated a patch, each eventType will have single async dispatcher thread. I will add unit test later. I think we first realize the multi thread dispatcher for each eventType, so that the event queue will not be a problem. And about the thread number, i counted the eventType number, it is in reasonable range. Then we can move to that, how can we consume the event queue faster. > RM multi-thread event processing mechanism > ------------------------------------------ > > Key: YARN-9927 > URL: https://issues.apache.org/jira/browse/YARN-9927 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn > Affects Versions: 3.0.0, 2.9.2 > Reporter: hcarrot > Assignee: Qi Zhu > Priority: Major > Attachments: RM multi-thread event processing mechanism.pdf, > YARN-9927.001.patch, YARN-9927.002.patch > > > Recently, we have observed serious event blocking in RM event dispatcher > queue. After analysis of RM event monitoring data and RM event processing > logic, we found that > 1) environment: a cluster with thousands of nodes > 2) RMNodeStatusEvent dominates 90% time consumption of RM event scheduler > 3) Meanwhile, RM event processing is in a single-thread mode, and It results > in the low headroom of RM event scheduler, thus performance of RM. > So we proposed a RM multi-thread event processing mechanism to improve RM > performance. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org