[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17260114#comment-17260114
]
Wangda Tan commented on YARN-10504:
-----------------------------------
[~gandras],
Thanks for updating this, your observation is correct: In the original patch I
moved calculation of absolute percentage from config parsing phase to refresh
queue phase. Why I did that?
- Because of the weight mode will impact relative percentage, queue parsing
phase is no longer able to get all percentages of queues in one shot. Even
though we can normalize weight values during parsing phase, however, if we have
dynamic queues under the same parent, parsing phase cannot get all percentage
correctly calcuated.
- The previous calculation of config properties are bit ad-hoc: Some values are
calculated during setupQueueConfigs, some values calculated in
reinitialization, some values calculated in updateClusterResource. The patch I
moved many of the config properties which needs runtime calculation (such as
absolute percentage) to updateCluserResources. (And removed from
setupQueueConfig)
- I noticed failure of AutoCreatedLeafQueue, but I didn't get a chance to dig
more for the root cause, sjince updateClusterResource will be called for
AutoCreatedLeafQueue, I don't know why it is still failing. Is it a test issue
or production issue? (Please note that updateClusterResource of CSQueue will be
called during CS init, reinit, and resource changes).
> Implement weight mode in Capacity Scheduler
> -------------------------------------------
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Benjamin Teke
> Assignee: Benjamin Teke
> Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
