[
https://issues.apache.org/jira/browse/YARN-10169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17251110#comment-17251110
]
zhuqi edited comment on YARN-10169 at 12/18/20, 1:49 AM:
---------------------------------------------------------
[~leftnoteasy] [~pbacsko], [~snemeth], [~sunilg], [~bteke]
When i write a unit test, to confirm that percentage below absolute, the queue
will always get wrong resource, 100 of the parent.
Becasue of the logic:
{code:java}
//Set absolute capacities for {capacity, maximum-capacity}
private static void updateAbsoluteCapacitiesByNodeLabels(
QueueCapacities queueCapacities, QueueCapacities parentQueueCapacities) {
for (String label : queueCapacities.getExistingNodeLabels()) {
float capacity = queueCapacities.getCapacity(label);
if (capacity > 0f) {
queueCapacities.setAbsoluteCapacity(
label,
capacity
* (parentQueueCapacities == null ? 1 : parentQueueCapacities
.getAbsoluteCapacity(label)));
}
float maxCapacity = queueCapacities.getMaximumCapacity(label);
if (maxCapacity > 0f) {
queueCapacities.setAbsoluteMaximumCapacity(
label,
maxCapacity
* (parentQueueCapacities == null ? 1 : parentQueueCapacities
.getAbsoluteMaximumCapacity(label)));
}
}
}
{code}
If we use absolute in parent, and also the capacity use the single absolute
resource. Here is the maxCapacity related code:
{code:java}
public float getNonLabeledQueueMaximumCapacity(String queue) {
String configuredCapacity = get(getQueuePrefix(queue) + MAXIMUM_CAPACITY);
boolean matcher = (configuredCapacity != null)
&& RESOURCE_PATTERN.matcher(configuredCapacity).find();
if (matcher) {
// Return capacity in percentage as 0 for non-root queues and 100 for
// root.From AbstractCSQueue, absolute resource will be parsed and
// updated. Once nodes are added/removed in cluster, capacity in
// percentage will also be re-calculated.
return 100.0f;
}
float maxCapacity = (configuredCapacity == null)
? MAXIMUM_CAPACITY_VALUE
: Float.parseFloat(configuredCapacity);
maxCapacity = (maxCapacity == DEFAULT_MAXIMUM_CAPACITY_VALUE)
? MAXIMUM_CAPACITY_VALUE
: maxCapacity;
return maxCapacity;
}
{code}
In capacity absolute resource mode, it will return 100.0f, then the maxCapacity
will be 100.
We should change it to support mixed mode，and if we now use mixed mode in
maxCapacity, it will not throw exception, but it will be wrong.
Also we should fix it in auto create queue.
was (Author: zhuqi):
[~leftnoteasy] [~pbacsko], [~snemeth], [~sunilg], [~bteke]
When i write a unit test, to confirm that percentage below absolute, the queue
will always get wrong resource, 100 of the parent.
Becasue of the logic:
{code:java}
//Set absolute capacities for {capacity, maximum-capacity}
private static void updateAbsoluteCapacitiesByNodeLabels(
QueueCapacities queueCapacities, QueueCapacities parentQueueCapacities) {
for (String label : queueCapacities.getExistingNodeLabels()) {
float capacity = queueCapacities.getCapacity(label);
if (capacity > 0f) {
queueCapacities.setAbsoluteCapacity(
label,
capacity
* (parentQueueCapacities == null ? 1 : parentQueueCapacities
.getAbsoluteCapacity(label)));
}
float maxCapacity = queueCapacities.getMaximumCapacity(label);
if (maxCapacity > 0f) {
queueCapacities.setAbsoluteMaximumCapacity(
label,
maxCapacity
* (parentQueueCapacities == null ? 1 : parentQueueCapacities
.getAbsoluteMaximumCapacity(label)));
}
}
}
{code}
If we use absolute in parent, and also the capacity use the single absolute
resource. The parent's parentQueueCapacities will be null, then the value will
be maxCapacity, here is the maxCapacity related code:
{code:java}
public float getNonLabeledQueueMaximumCapacity(String queue) {
String configuredCapacity = get(getQueuePrefix(queue) + MAXIMUM_CAPACITY);
boolean matcher = (configuredCapacity != null)
&& RESOURCE_PATTERN.matcher(configuredCapacity).find();
if (matcher) {
// Return capacity in percentage as 0 for non-root queues and 100 for
// root.From AbstractCSQueue, absolute resource will be parsed and
// updated. Once nodes are added/removed in cluster, capacity in
// percentage will also be re-calculated.
return 100.0f;
}
float maxCapacity = (configuredCapacity == null)
? MAXIMUM_CAPACITY_VALUE
: Float.parseFloat(configuredCapacity);
maxCapacity = (maxCapacity == DEFAULT_MAXIMUM_CAPACITY_VALUE)
? MAXIMUM_CAPACITY_VALUE
: maxCapacity;
return maxCapacity;
}
{code}
In capacity absolute resource mode, it will return 100.0f, then the maxCapacity
will be 100.
We should change it to support mixed mode，and if we now use mixed mode in
maxCapacity, it will not throw exception, but it will be wrong.
Also we should fix it in auto create queue.
> Mixed absolute resource value and percentage-based resource value in
> CapacityScheduler should fail
> --------------------------------------------------------------------------------------------------
>
> Key: YARN-10169
> URL: https://issues.apache.org/jira/browse/YARN-10169
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Assignee: zhuqi
> Priority: Blocker
> Attachments: YARN-10169.001.patch, YARN-10169.002.patch,
> YARN-10169.003.patch
>
>
> To me this is a bug: if there's a queue has capacity set to float, and
> maximum-capacity set to absolute value. Existing logic allows the behavior.
> For example:
> {code:java}
> queue.capacity = 0.8
> queue.maximum-capacity = [mem=x, vcore=y] {code}
> We should throw exception when configured like this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
