[jira] [Commented] (YARN-3689) FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should change position when compare priority
[ https://issues.apache.org/jira/browse/YARN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694829#comment-14694829 ] zhoulinlin commented on YARN-3689: -- Thanks! I thought "Higher Interger indicates Lower priority", so I am wrong. The current code is fine. > FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, > the "s1" and "s2" should change position when compare priority > - > > Key: YARN-3689 > URL: https://issues.apache.org/jira/browse/YARN-3689 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler, scheduler >Affects Versions: 2.5.0 >Reporter: zhoulinlin >Assignee: Ajith S > > In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should > change position when compare priority. > I did a test. Configured the schedulerpolicy "fifo", submitted 2 jobs to the > same queue. > The result is below: > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > before sort -- > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0001 appPririty:4 > appStartTime:1432094170038 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0002 appPririty:2 > appStartTime:1432094173131 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > after sort % > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0001 appPririty:4 > appStartTime:1432094170038 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0002 appPririty:2 > appStartTime:1432094173131 > But when change the "s1" and "s2" position like below: > public int compare(Schedulable s1, Schedulable s2) { > int res = s2.getPriority().compareTo(s1.getPriority()); > .} > The result: > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > before sort -- > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0009 appPririty:4 > appStartTime:1432092992503 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0010 appPririty:2 > appStartTime:1432092996437 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > after sort % > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0010 appPririty:2 > appStartTime:1432092996437 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0009 appPririty:4 > appStartTime:1432092992503 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3689) FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should change position when compare priority
[ https://issues.apache.org/jira/browse/YARN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694784#comment-14694784 ] Ajith S commented on YARN-3689: --- Please ignore previous comment, the current code is fine if {quote} *Higher Integer* indicates *Higher Priority* {quote} i overlooked it :) > FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, > the "s1" and "s2" should change position when compare priority > - > > Key: YARN-3689 > URL: https://issues.apache.org/jira/browse/YARN-3689 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler, scheduler >Affects Versions: 2.5.0 >Reporter: zhoulinlin >Assignee: Ajith S > > In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should > change position when compare priority. > I did a test. Configured the schedulerpolicy "fifo", submitted 2 jobs to the > same queue. > The result is below: > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > before sort -- > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0001 appPririty:4 > appStartTime:1432094170038 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0002 appPririty:2 > appStartTime:1432094173131 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > after sort % > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0001 appPririty:4 > appStartTime:1432094170038 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0002 appPririty:2 > appStartTime:1432094173131 > But when change the "s1" and "s2" position like below: > public int compare(Schedulable s1, Schedulable s2) { > int res = s2.getPriority().compareTo(s1.getPriority()); > .} > The result: > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > before sort -- > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0009 appPririty:4 > appStartTime:1432092992503 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0010 appPririty:2 > appStartTime:1432092996437 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > after sort % > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0010 appPririty:2 > appStartTime:1432092996437 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0009 appPririty:4 > appStartTime:1432092992503 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3689) FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should change position when compare priority
[ https://issues.apache.org/jira/browse/YARN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694771#comment-14694771 ] Ajith S commented on YARN-3689: --- Hi [~rohithsharma] Thanks for the input. {quote} *Higher Integer* indicates *Higher priority* {quote} So, i think we should flip the operands in org.apache.hadoop.yarn.api.records.Priority.compareTo(Priority) > FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, > the "s1" and "s2" should change position when compare priority > - > > Key: YARN-3689 > URL: https://issues.apache.org/jira/browse/YARN-3689 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler, scheduler >Affects Versions: 2.5.0 >Reporter: zhoulinlin >Assignee: Ajith S > > In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should > change position when compare priority. > I did a test. Configured the schedulerpolicy "fifo", submitted 2 jobs to the > same queue. > The result is below: > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > before sort -- > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0001 appPririty:4 > appStartTime:1432094170038 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0002 appPririty:2 > appStartTime:1432094173131 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > after sort % > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0001 appPririty:4 > appStartTime:1432094170038 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0002 appPririty:2 > appStartTime:1432094173131 > But when change the "s1" and "s2" position like below: > public int compare(Schedulable s1, Schedulable s2) { > int res = s2.getPriority().compareTo(s1.getPriority()); > .} > The result: > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > before sort -- > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0009 appPririty:4 > appStartTime:1432092992503 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0010 appPririty:2 > appStartTime:1432092996437 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > after sort % > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0010 appPririty:2 > appStartTime:1432092996437 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0009 appPririty:4 > appStartTime:1432092992503 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3689) FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should change position when compare priority
[ https://issues.apache.org/jira/browse/YARN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694687#comment-14694687 ] Rohith Sharma K S commented on YARN-3689: - As per the application priority design, *Higher Interger* indicates *Higher priority*, so comparator implementation seems to be fine for me. And the test by [~ajithshetty] also proving higher prioirty value i.e 2 is first in the list. > FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, > the "s1" and "s2" should change position when compare priority > - > > Key: YARN-3689 > URL: https://issues.apache.org/jira/browse/YARN-3689 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler, scheduler >Affects Versions: 2.5.0 >Reporter: zhoulinlin >Assignee: Ajith S > > In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should > change position when compare priority. > I did a test. Configured the schedulerpolicy "fifo", submitted 2 jobs to the > same queue. > The result is below: > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > before sort -- > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0001 appPririty:4 > appStartTime:1432094170038 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0002 appPririty:2 > appStartTime:1432094173131 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > after sort % > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0001 appPririty:4 > appStartTime:1432094170038 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0002 appPririty:2 > appStartTime:1432094173131 > But when change the "s1" and "s2" position like below: > public int compare(Schedulable s1, Schedulable s2) { > int res = s2.getPriority().compareTo(s1.getPriority()); > .} > The result: > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > before sort -- > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0009 appPririty:4 > appStartTime:1432092992503 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0010 appPririty:2 > appStartTime:1432092996437 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > after sort % > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0010 appPririty:2 > appStartTime:1432092996437 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0009 appPririty:4 > appStartTime:1432092992503 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3689) FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should change position when compare priority
[ https://issues.apache.org/jira/browse/YARN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694661#comment-14694661 ] Ajith S commented on YARN-3689: --- I think it follows the contract of compare, but as Collections.sort is processed, in case of priority its suppose to negate. just flipping the s1 and s2 may serve the purpose, but violate compare contract > FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, > the "s1" and "s2" should change position when compare priority > - > > Key: YARN-3689 > URL: https://issues.apache.org/jira/browse/YARN-3689 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler, scheduler >Affects Versions: 2.5.0 >Reporter: zhoulinlin >Assignee: Ajith S > > In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should > change position when compare priority. > I did a test. Configured the schedulerpolicy "fifo", submitted 2 jobs to the > same queue. > The result is below: > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > before sort -- > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0001 appPririty:4 > appStartTime:1432094170038 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0002 appPririty:2 > appStartTime:1432094173131 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > after sort % > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0001 appPririty:4 > appStartTime:1432094170038 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0002 appPririty:2 > appStartTime:1432094173131 > But when change the "s1" and "s2" position like below: > public int compare(Schedulable s1, Schedulable s2) { > int res = s2.getPriority().compareTo(s1.getPriority()); > .} > The result: > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > before sort -- > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0009 appPririty:4 > appStartTime:1432092992503 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0010 appPririty:2 > appStartTime:1432092996437 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > after sort % > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0010 appPririty:2 > appStartTime:1432092996437 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0009 appPririty:4 > appStartTime:1432092992503 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3689) FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should change position when compare priority
[ https://issues.apache.org/jira/browse/YARN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694655#comment-14694655 ] Ajith S commented on YARN-3689: --- Hi [~lyn] Thanks for reporting the issue. *org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.FifoPolicy.FifoComparator.compare(Schedulable, Schedulable)* seems fine to me as the lower the priority number, the schedule holds higher value. But i think the problem is in Collection.sort in FSLeafQueue which will order the elements in ascending order. Any thoughts.? Here is a small test case {code} //create 2 schedule with priority Schedulable s1 = createSchedulable(1); Schedulable s2 = createSchedulable(2); FifoPolicy fp = new FifoPolicy(); Comparator fpComparator = fp.getComparator(); int compare = fpComparator.compare(s1, s2); //ensure smaller the number, is considered higher assertTrue(compare > 0); List list = new ArrayList<>(); list.add(s1); list.add(s2); Collections.sort(list,fpComparator); assertEquals(s1,list.get(0)); //fails assertEquals(s2,list.get(1)); //fails {code} > FifoComparator logic is wrong. In method "compare" in "FifoPolicy.java" file, > the "s1" and "s2" should change position when compare priority > - > > Key: YARN-3689 > URL: https://issues.apache.org/jira/browse/YARN-3689 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler, scheduler >Affects Versions: 2.5.0 >Reporter: zhoulinlin >Assignee: Ajith S > > In method "compare" in "FifoPolicy.java" file, the "s1" and "s2" should > change position when compare priority. > I did a test. Configured the schedulerpolicy "fifo", submitted 2 jobs to the > same queue. > The result is below: > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > before sort -- > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0001 appPririty:4 > appStartTime:1432094170038 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0002 appPririty:2 > appStartTime:1432094173131 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > after sort % > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0001 appPririty:4 > appStartTime:1432094170038 > 2015-05-20 11:57:41,449 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432094103221_0002 appPririty:2 > appStartTime:1432094173131 > But when change the "s1" and "s2" position like below: > public int compare(Schedulable s1, Schedulable s2) { > int res = s2.getPriority().compareTo(s1.getPriority()); > .} > The result: > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > before sort -- > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0009 appPririty:4 > appStartTime:1432092992503 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0010 appPririty:2 > appStartTime:1432092996437 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > after sort % > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0010 appPririty:2 > appStartTime:1432092996437 > 2015-05-20 11:36:37,119 DEBUG > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: > appName:application_1432090734333_0009 appPririty:4 > appStartTime:1432092992503 -- This message was sent by Atlassian JIRA (v6.3.4#6332)