[jira] [Commented] (YARN-6516) FairScheduler:the algorithm of assignContainer is so slow for it only can assign a thousand containers per second
[ https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984016#comment-15984016 ] JackZhou commented on YARN-6516: [~yufeigu] Thank you for you answers! 1. I was saw it in ganglia for "assigning a container took 3ms" , and it think the metric is real. Because we have about 80 queues and the two active queue have about 1000 applications. Use the algorithm to assign a container, it cost 3ms is Certainly need！ 2. You are right for "The 39 containers are for AM", and I am pressure test RM fair scheduler for improve the performance of schedule. But it is not important for the container is used for AM or a normal container, it is important that the resource is very leisure but the scheduler schedule is so slow. 3. I saw many jira talk about remove continuous scheduling, but I have not saw any report about the performance decrease when use continuous scheduling. In a word, I think there is too mush space for optimization especially in scheduler. > FairScheduler:the algorithm of assignContainer is so slow for it only can > assign a thousand containers per second > - > > Key: YARN-6516 > URL: https://issues.apache.org/jira/browse/YARN-6516 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.7.2 >Reporter: JackZhou > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6516) FairScheduler:the algorithm of assignContainer is so slow for it only can assign a thousand containers per second
[ https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983443#comment-15983443 ] Yufei Gu commented on YARN-6516: IIUC, there were only 40 containers in your cluster over time. More likely it is not a performance issue. Some thoughts and questions: # Continuous scheduling is unnecessary for a cluster with 2500 nodes. Turn on the assignmultiple should be enough. # There were 39 containers for 39 apps in your cluster. I guess these containers are all for AM. No tasks for apps? That's impossible for MR jobs. # How did you conclude "assigning a container took 3ms"? > FairScheduler:the algorithm of assignContainer is so slow for it only can > assign a thousand containers per second > - > > Key: YARN-6516 > URL: https://issues.apache.org/jira/browse/YARN-6516 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.7.2 >Reporter: JackZhou > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6516) FairScheduler:the algorithm of assignContainer is so slow for it only can assign a thousand containers per second
[ https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982543#comment-15982543 ] JackZhou commented on YARN-6516: [~piaoyu zhang] How mush performance do the two patch can improve ? > FairScheduler:the algorithm of assignContainer is so slow for it only can > assign a thousand containers per second > - > > Key: YARN-6516 > URL: https://issues.apache.org/jira/browse/YARN-6516 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.7.2 >Reporter: JackZhou > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6516) FairScheduler:the algorithm of assignContainer is so slow for it only can assign a thousand containers per second
[ https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982465#comment-15982465 ] zhangyubiao commented on YARN-6516: --- Would you like try YARN-5483 and YARN-3139 patch. We have the same problem before. > FairScheduler:the algorithm of assignContainer is so slow for it only can > assign a thousand containers per second > - > > Key: YARN-6516 > URL: https://issues.apache.org/jira/browse/YARN-6516 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.7.2 >Reporter: JackZhou > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6516) FairScheduler:the algorithm of assignContainer is so slow for it only can assign a thousand containers per second
[ https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982319#comment-15982319 ] JackZhou commented on YARN-6516: [~yufeigu] I am test in a real cluster which have about 2500 nodes. I have already set continuous scheduling on but I set the yarn.scheduler.fair.continuous-scheduling-sleep-ms to 500, so it is run per 500 ms. There is about 80 parent queues in my scheduler and about 200 queues total. I think the scheduler assign a thousand containers per second is a pretty ideal scenario, because if my queue is very empty it will cost 1ms to assign a container for scheduler. But in my test, I have two queues, the queue information as blow: Used Resources: Num Active Applications:19 Num Pending Applications: 1057 Min Resources: Max Resources: Max Running Applications: 4000 Steady Fair Share: Instantaneous Fair Share: Used Resources: Num Active Applications:20 Num Pending Applications: 781 Min Resources: Max Resources: Max Running Applications: 4000 Steady Fair Share: Instantaneous Fair Share: The cost to assign a container up to about 3ms, and the scheduler only can scheduler about 40 containers. It is so slow! Apps Submitted Apps PendingApps RunningApps Completed Containers Running Memory Used Memory TotalMemory Reserved VCores Used VCores TotalVCores Reserved Active NodesDecommissioned NodesLost Nodes Unhealthy Nodes Rebooted Nodes 10268 183839 839139 39 GB 95 TB 0 B 39 97280 0 24322 64 0 0 User Metrics for hadoop > FairScheduler:the algorithm of assignContainer is so slow for it only can > assign a thousand containers per second > - > > Key: YARN-6516 > URL: https://issues.apache.org/jira/browse/YARN-6516 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.7.2 >Reporter: JackZhou > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6516) FairScheduler:the algorithm of assignContainer is so slow for it only can assign a thousand containers per second
[ https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981666#comment-15981666 ] Rushabh S Shah commented on YARN-6516: -- [~zhouyunfan]: do you mind changing your visible name ? > FairScheduler:the algorithm of assignContainer is so slow for it only can > assign a thousand containers per second > - > > Key: YARN-6516 > URL: https://issues.apache.org/jira/browse/YARN-6516 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.7.2 >Reporter: GirlKiller > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6516) FairScheduler:the algorithm of assignContainer is so slow for it only can assign a thousand containers per second
[ https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981478#comment-15981478 ] Yufei Gu commented on YARN-6516: Can you provide more details about test environment? Is this tested in a real cluster? How many nodes are there? Was continuous scheduling on? What exact version did you test? Besides, I don't think it is blocker. > FairScheduler:the algorithm of assignContainer is so slow for it only can > assign a thousand containers per second > - > > Key: YARN-6516 > URL: https://issues.apache.org/jira/browse/YARN-6516 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.7.2 >Reporter: GirlKiller >Priority: Blocker > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6516) FairScheduler:the algorithm of assignContainer is so slow for it only can assign a thousand containers per second
[ https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15980852#comment-15980852 ] GirlKiller commented on YARN-6516: -- is the avg value > FairScheduler:the algorithm of assignContainer is so slow for it only can > assign a thousand containers per second > - > > Key: YARN-6516 > URL: https://issues.apache.org/jira/browse/YARN-6516 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.7.2 >Reporter: GirlKiller > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6516) FairScheduler:the algorithm of assignContainer is so slow for it only can assign a thousand containers per second
[ https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15980777#comment-15980777 ] zhangyubiao commented on YARN-6516: --- a thousand containers per second is the max value or the avg value? > FairScheduler:the algorithm of assignContainer is so slow for it only can > assign a thousand containers per second > - > > Key: YARN-6516 > URL: https://issues.apache.org/jira/browse/YARN-6516 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.7.2 >Reporter: GirlKiller > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
