[ https://issues.apache.org/jira/browse/YARN-5188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15309375#comment-15309375 ]
ChenFolin edited comment on YARN-5188 at 6/8/16 2:09 AM: --------------------------------------------------------- Thanks Xianyin Xin, TreeSet oparate time complexity here is : O(log1) + O(log2) + ... + O(log(app num)), and binary search O(log(app num)). point 2 is the same with YARN-4090. was (Author: chenfolin): Thanks Xianyin Xin, TreeSet oparate time complexity here is : O(log1) + O(log2) + ... + O(log(app num)), and binary search O(log(app num)). point 2 is the same with YARN-4090. And can you tell me your phone number or wechat or qq ? Send a mail to chenfo...@gmail.com > FairScheduler performance bug > ----------------------------- > > Key: YARN-5188 > URL: https://issues.apache.org/jira/browse/YARN-5188 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler > Affects Versions: 2.5.0 > Reporter: ChenFolin > Attachments: YARN-5188-1.patch > > > My Hadoop Cluster has recently encountered a performance problem. Details as > Follows. > There are two point which can cause this performance issue. > 1: application sort before assign container at FSLeafQueue. TreeSet is not > the best, Why not keep orderly ? and then we can use binary search to help > keep orderly when a application's resource usage has changed. > 2: queue sort and assignContainerPreCheck will lead to compute all leafqueue > resource usage ,Why can we store the leafqueue usage at memory and update it > when assign container op release container happen? > > The efficiency of assign container in the Resourcemanager may fall > when the number of running and pending application grows. And the fact is the > cluster has too many PendingMB or PengdingVcore , and the Cluster > current utilization rate may below 20%. > I checked the resourcemanager logs, I found that every assign > container may cost 5 ~ 10 ms, but just 0 ~ 1 ms at usual time. > > I use TestFairScheduler to reproduce the scene: > > Just one queue: root.defalut > 10240 apps. > > assign container avg time: 6753.9 us ( 6.7539 ms) > apps sort time (FSLeafQueue : Collections.sort(runnableApps, > comparator); ): 4657.01 us ( 4.657 ms ) > compute LeafQueue Resource usage : 905.171 us ( 0.905171 ms ) > > When just root.default, one assign container op contains : ( one apps > sort op ) + 2 * ( compute leafqueue usage op ) > According to the above situation, I think the assign container op has > a performance problem . -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org