[
https://issues.apache.org/jira/browse/HADOOP-5003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662622#action_12662622
]
Vivek Ratan commented on HADOOP-5003:
-------------------------------------
bq. ... I believe the current algorithm is more stable ...
How so?
If we round down, we can end up with the situation where the sum of GCs is
slightly less than the overall cluster capacity. If we 'round up' (note that my
suggestion of using Math.round() doesn't always round up; it only rounds up if
the decimal value is 0.50 or greater, and rounds down otherwise), the sum of
GCs can be slightly more than the overall cluster capacity. Neither of these
situations affects the stability of the algorithm. GCs are used to sort the
queues when deciding the order in which queues are considered for a TT. GCs are
also used to decide when, and how much, to reclaim capacity. The difference
between rounding up and down a queue's GC is at at most 1 slot. That really
shouldn't make a difference (hence I marked this issue as 'Minor').
My only argument for this Jira is that rounding sounds more fair than casting
from a float to an int. (BTW, your example is not quite right. With 10 slots,
and 33% capacity, each queue still gets 3 slots, whether you use Math.round(),
as I am proposing, or whether you used an int cast. But I think i get your
point. If the # of slots is 11, each queue's GC will be 4 with the former,and 3
with the latter.) Math.round() doesn't always round up - it rounds up anything
over, and equal to, 0.50, and rounds down anything below 0.50. That seems
fairer than always rounding down. If a queue's GC% is 45%,and the # of slots is
4, it seems fairer to give that queue a GC of 2, rather than a GC of 1.
> When computing absoluet guaranteed capacity (GC) from a percent value,
> Capacity Scheduler should round up floats, rather than truncate them.
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-5003
> URL: https://issues.apache.org/jira/browse/HADOOP-5003
> Project: Hadoop Core
> Issue Type: Bug
> Reporter: Vivek Ratan
> Priority: Minor
>
> The Capacity Scheduler calculates a queue's absolute GC value by getting its
> percent of the total cluster capacity (which is a float, since the configured
> GC% is a float) and casting it to an int. Casting a float to an int always
> rounds down. For very small clusters, this can result in the GC of a queue
> being one lower than what it should be. For example, if Q1 has a GC of 50%,
> Q2 has a GC of 40%, and Q3 has a GC of 10%, and if the cluster capacity is 4
> (as we have, in our test cases), Q1's GC works out to 2, Q2's to 1, and Q3's
> to 0 with today's code. Q2's capacity should really be 2, as 40% of 4,
> rounded up, should be 2.
> Simple fix is to use Math.round() rather than cast to an int.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
