[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511068#comment-14511068 ] Hudson commented on YARN-3319: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2123 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2123/]) YARN-3319. Implement a FairOrderingPolicy. (Craig Welch via wangda) (wangda: rev 395205444e8a9ae6fc86f0a441e98486a775511a) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/CompoundComparator.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FairOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch, YARN-3319.74.patch, YARN-3319.75.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511029#comment-14511029 ] Hudson commented on YARN-3319: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #174 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/174/]) YARN-3319. Implement a FairOrderingPolicy. (Craig Welch via wangda) (wangda: rev 395205444e8a9ae6fc86f0a441e98486a775511a) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FairOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/CompoundComparator.java > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch, YARN-3319.74.patch, YARN-3319.75.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14510926#comment-14510926 ] Hudson commented on YARN-3319: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #907 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/907/]) YARN-3319. Implement a FairOrderingPolicy. (Craig Welch via wangda) (wangda: rev 395205444e8a9ae6fc86f0a441e98486a775511a) * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FairOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/CompoundComparator.java > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch, YARN-3319.74.patch, YARN-3319.75.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14510887#comment-14510887 ] Hudson commented on YARN-3319: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #173 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/173/]) YARN-3319. Implement a FairOrderingPolicy. (Craig Welch via wangda) (wangda: rev 395205444e8a9ae6fc86f0a441e98486a775511a) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/CompoundComparator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FairOrderingPolicy.java * hadoop-yarn-project/CHANGES.txt > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch, YARN-3319.74.patch, YARN-3319.75.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14510868#comment-14510868 ] Hudson commented on YARN-3319: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #164 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/164/]) YARN-3319. Implement a FairOrderingPolicy. (Craig Welch via wangda) (wangda: rev 395205444e8a9ae6fc86f0a441e98486a775511a) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/CompoundComparator.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FairOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch, YARN-3319.74.patch, YARN-3319.75.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14510851#comment-14510851 ] Hudson commented on YARN-3319: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2105 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2105/]) YARN-3319. Implement a FairOrderingPolicy. (Craig Welch via wangda) (wangda: rev 395205444e8a9ae6fc86f0a441e98486a775511a) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FairOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/CompoundComparator.java * hadoop-yarn-project/CHANGES.txt > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch, YARN-3319.74.patch, YARN-3319.75.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509641#comment-14509641 ] Craig Welch commented on YARN-3319: --- Yes, it's configured in the capacity scheduler configuration with something like this: {code} (yarn-queue-prefix).ordering-policy.fair.enable-size-based-weight true {code} > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch, YARN-3319.74.patch, YARN-3319.75.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509630#comment-14509630 ] Vinod Kumar Vavilapalli commented on YARN-3319: --- Is FairOrderingPolicy.ENABLE_SIZE_BASED_WEIGHT supposed to be admin visible? If so, we need a better, fully qualified name.. > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch, YARN-3319.74.patch, YARN-3319.75.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509470#comment-14509470 ] Hudson commented on YARN-3319: -- FAILURE: Integrated in Hadoop-trunk-Commit #7648 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7648/]) YARN-3319. Implement a FairOrderingPolicy. (Craig Welch via wangda) (wangda: rev 395205444e8a9ae6fc86f0a441e98486a775511a) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FairOrderingPolicy.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/CompoundComparator.java > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch, YARN-3319.74.patch, YARN-3319.75.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507920#comment-14507920 ] Wangda Tan commented on YARN-3319: -- Retriggerred Jenkins to see if problem still exists. > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch, YARN-3319.74.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507917#comment-14507917 ] Craig Welch commented on YARN-3319: --- The failed tests pass on my box with the patch, unrelated. The checkstyle is referring to ResourceLimits, which the patch doesn't change... poking around in the build artifacts there are some exceptions in some of the checkstyle stuff, I'm not sure it's actually working correctly > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch, YARN-3319.74.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507873#comment-14507873 ] Hadoop QA commented on YARN-3319: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 14m 59s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 2 new or modified test files. | | {color:red}-1{color} | whitespace | 0m 0s | The patch has 99 line(s) that end in whitespace. | | {color:green}+1{color} | javac | 7m 45s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 5m 25s | The applied patch generated 3 additional checkstyle issues. | | {color:green}+1{color} | install | 1m 33s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 1m 17s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. | | {color:red}-1{color} | yarn tests | 50m 50s | Tests failed in hadoop-yarn-server-resourcemanager. | | | | 92m 21s | | \\ \\ || Reason || Tests || | Failed unit tests | hadoop.yarn.server.resourcemanager.TestRM | | | hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA | | | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12727310/YARN-3319.74.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / 12f4df0 | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/7454/artifact/patchprocess/whitespace.txt | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/7454/artifact/patchprocess/checkstyle-result-diff.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/7454/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/7454/testReport/ | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7454//console | This message was automatically generated. > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch, YARN-3319.74.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507848#comment-14507848 ] Wangda Tan commented on YARN-3319: -- Thanks for update, [~cwelch]. Patch LGTM, +1. > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch, YARN-3319.74.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14507701#comment-14507701 ] Craig Welch commented on YARN-3319: --- bq. Some minor comments about configuration part by index: 1) done 2) done 3) done - see below bq. Do you think is it better to make property in .ordering-policy..?... Now that there is not proper composition only one policy can be active at a time and it shouldn't be necessary to namespace config items this way. At the same time, I could see us getting back to proper composition at some point, where this would be helpful. I've implemented it as a prefix convention in the policy instead of constraining the contents of the map in the capacity scheduler configuration. This is because we still support passing a class name as the policy type, which would make the configurations for "class name based" items unwieldy. It would also allow us to have shared configuration items between policies if we do end up with proper composition again. The end result of the configuration was as you suggested 4) done 5) done bq. FairOrderingPolicy: all 3 done bq. Findbugs warning? Failed to stage change, so it didn't make it into patch, should be there now. > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505678#comment-14505678 ] Wangda Tan commented on YARN-3319: -- Some minor comments about configuration part: 1) getting config mapping should be a generic method, you can use Configuration.iterator() to get all properties of a policy. 2) Policy will handle properties such as "size-based-weight" in its own configure(..) 3) Do you think is it better to make property in .ordering-policy..? In your example, "sizeBasedWeight" more like a property of fair policy instead of others. I perfer add a 4) "sizeBasedWeight" to "size-based-weight-enabled" (or enable-size-based-weight) to make it consistent with other configurations of CS. Add the enable to indicate it's a boolean instead of float. 5) Make the String="size-based-weight" to a final static variable like other configuration keys. Same of "fifo" and "fifo". Test should use these final static variables as well FairOrderingPolicy: 1) Mark get/setSizeBasedWeight to VisibleForTesting? 2) getMagnitude should be private 3) LOG is not used by anybody, could be removed. Findbugs warning? > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505421#comment-14505421 ] Hadoop QA commented on YARN-3319: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726913/YARN-3319.73.patch against trunk revision 8ddbb8d. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7426//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7426//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7426//console This message is automatically generated. > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, > YARN-3319.72.patch, YARN-3319.73.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504176#comment-14504176 ] Hadoop QA commented on YARN-3319: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726734/YARN-3319.72.patch against trunk revision 44872b7. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7413//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7413//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7413//console This message is automatically generated. > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch, YARN-3319.53.patch, > YARN-3319.58.patch, YARN-3319.70.patch, YARN-3319.71.patch, YARN-3319.72.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3319) Implement a FairOrderingPolicy
[ https://issues.apache.org/jira/browse/YARN-3319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14484368#comment-14484368 ] Craig Welch commented on YARN-3319: --- Apply after applying YARN-3318 and YARN-3463 > Implement a FairOrderingPolicy > -- > > Key: YARN-3319 > URL: https://issues.apache.org/jira/browse/YARN-3319 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.13.patch, YARN-3319.14.patch, > YARN-3319.17.patch, YARN-3319.35.patch, YARN-3319.39.patch, > YARN-3319.45.patch, YARN-3319.47.patch > > > Implement a FairOrderingPolicy which prefers to allocate to > SchedulerProcesses with least current usage, very similar to the > FairScheduler's FairSharePolicy. > The Policy will offer allocations to applications in a queue in order of > least resources used, and preempt applications in reverse order (from most > resources used). This will include conditional support for sizeBasedWeight > style adjustment > Optionally, based on a conditional configuration to enable sizeBasedWeight > (default false), an adjustment to boost larger applications (to offset the > natural preference for smaller applications) will adjust the resource usage > value based on demand, dividing it by the below value: > Math.log1p(app memory demand) / Math.log(2); > In cases where the above is indeterminate (two applications are equal after > this comparison), behavior falls back to comparison based on the application > id, which is generally lexically FIFO for that comparison -- This message was sent by Atlassian JIRA (v6.3.4#6332)