[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661616#comment-16661616 ] Chen Yufei commented on YARN-8513: -- Our YARN cluster is now working fine. Thanks [~cheersyang]

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654575#comment-16654575 ] Weiwei Yang commented on YARN-8513: --- Thanks [~leftnoteasy], [~hustnn], I created YARN-8896 to track this

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652886#comment-16652886 ] niu commented on YARN-8513: --- [~cheersyang] [~leftnoteasy] Thanks. > CapacityScheduler infinite loop when queue

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652645#comment-16652645 ] Wangda Tan commented on YARN-8513: -- Sounds like a plan, default value set to 100 may make more sense.

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651349#comment-16651349 ] Weiwei Yang commented on YARN-8513: --- Hi [~leftnoteasy]/[~cyfdecyf]/[~hustnn]/[~Tao Yang] When this

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651024#comment-16651024 ] Chen Yufei commented on YARN-8513: -- Make some corrections to my previous comment.   The behavior of

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649797#comment-16649797 ] Chen Yufei commented on YARN-8513: -- [~Tao Yang] I encounter this problem again today. I see logs saying

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623040#comment-16623040 ] Tao Yang commented on YARN-8513: Hi, [~cyfdecyf] Dose your cluster have empty resource type? This problem

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611742#comment-16611742 ] Chen Yufei commented on YARN-8513: -- [~cheersyang] Thanks for your help. I haven't run into any problem

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611677#comment-16611677 ] Weiwei Yang commented on YARN-8513: --- Hi [~cyfdecyf] Did the config changes I suggested last time

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610141#comment-16610141 ] Chen Yufei commented on YARN-8513: -- [~leftnoteasy] Thanks for taking time to investigate this.    The

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608263#comment-16608263 ] niu commented on YARN-8513: --- [~leftnoteasy] . No problem. Take your time. Let me try first. > CapacityScheduler

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608213#comment-16608213 ] Wangda Tan commented on YARN-8513: -- [~hustnn], I agree that it is still a problem, but relatively minor

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607901#comment-16607901 ] niu commented on YARN-8513: --- Thanks [~leftnoteasy] for your effort to look at this problem. In my attached

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607495#comment-16607495 ] Wangda Tan commented on YARN-8513: -- Spent good amount of time to check the issue. I found scheduler

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602522#comment-16602522 ] niu commented on YARN-8513: --- Debug dump: {code:java} 2018-09-03 11:44:11,175 DEBUG

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602519#comment-16602519 ] niu commented on YARN-8513: --- Attached the debug log. > CapacityScheduler infinite loop when queue is near fully

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602507#comment-16602507 ] niu commented on YARN-8513: --- I also tested. This happens when node label is not used. Does it cause by the

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602326#comment-16602326 ] Wangda Tan commented on YARN-8513: -- And btw, I found a comment in LeafQueue: {code:java} private void

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602323#comment-16602323 ] Wangda Tan commented on YARN-8513: -- Interesting, it must be caused by CS allocation doesn't fully

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601223#comment-16601223 ] niu commented on YARN-8513: --- Thank. I will try it tomorrow. > CapacityScheduler infinite loop when queue is

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601171#comment-16601171 ] Janus Chow commented on YARN-8513: -- [~leftnoteasy] Can reproduce as follow: 1.Specify 2 queues:     -

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601138#comment-16601138 ] Janus Chow commented on YARN-8513: -- Set two queues: - queue1:     - capacity: 50     - maximum-capacity:

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593141#comment-16593141 ] niu commented on YARN-8513: --- OK. I will try. > CapacityScheduler infinite loop when queue is near fully

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591878#comment-16591878 ] Wangda Tan commented on YARN-8513: -- [~hustnn], what is the cause of "Failed to accept allocation

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591337#comment-16591337 ] niu commented on YARN-8513: --- I also check the logic for printing out of the error "Failed to accept allocation

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588238#comment-16588238 ] Chen Yufei commented on YARN-8513: -- [~leftnoteasy] My original config did not have the two config options

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16586854#comment-16586854 ] Wangda Tan commented on YARN-8513: -- Interesting, [~cheersyang], I can only think about reservation

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16586811#comment-16586811 ] Weiwei Yang commented on YARN-8513: --- Discussed with [~cyfdecyf] in the slack channel, it looks like this

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16585359#comment-16585359 ] Weiwei Yang commented on YARN-8513: --- Thanks for the info [~cyfdecyf]. To work more efficiently on this

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16585036#comment-16585036 ] Chen Yufei commented on YARN-8513: -- [~cheersyang] Thanks for looking into this issue. The log message

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584825#comment-16584825 ] Weiwei Yang commented on YARN-8513: --- Hi [~cyfdecyf] >From the RM log you uploaded, in 1 sec, there are

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584787#comment-16584787 ] Chen Yufei commented on YARN-8513: -- New jstack/top and RM logs are uploaded and prefixed with yarn3. We

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583270#comment-16583270 ] Chen Yufei commented on YARN-8513: -- [~leftnoteasy] Thanks for your help. I'll enable capacity scheduler

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583026#comment-16583026 ] Wangda Tan commented on YARN-8513: -- [~cyfdecyf], Could u upload logs/jstacks for 3.1.0 deployment? We

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582584#comment-16582584 ] Chen Yufei commented on YARN-8513: -- [~leftnoteasy] We encounter the same problem twice today with Hadoop

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580862#comment-16580862 ] Chen Yufei commented on YARN-8513: -- We got infinite loops two times recently with 2.9.1, restarting

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552536#comment-16552536 ] niu commented on YARN-8513: --- We also met this problem in 2.9.1. > CapacityScheduler infinite loop when queue is

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551551#comment-16551551 ] Yuanbo Liu commented on YARN-8513: -- Sorry for the late response. Quite busy this week. I will go through

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544742#comment-16544742 ] Chen Yufei commented on YARN-8513: -- [~yuanbo] I've uploaded jstack and top log when the problem appears

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544234#comment-16544234 ] Chen Yufei commented on YARN-8513: -- [~leftnoteasy] Thanks for your suggestions. I'll deploy a test

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544231#comment-16544231 ] Chen Yufei commented on YARN-8513: -- [~yuanbo] I'll capture those info when I encounter this problem again

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16543689#comment-16543689 ] Wangda Tan commented on YARN-8513: -- [~cyfdecyf], I couldn't find the error message on the latest

[jira] [Commented] (YARN-8513) CapacityScheduler infinite loop when queue is near fully utilized

[ https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16542719#comment-16542719 ] Yuanbo Liu commented on YARN-8513: -- [~cyfdecyf] Can you reproduce this issue and capture the stack of RM