[
https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suma Shivaprasad updated YARN-7473:
-----------------------------------
Attachment: YARN-7473.11.patch
Thanks [~sunilg] and [~wangda] Attaching patch with most of the review comments
addressed
{quote} Once queue capacity is set to 0, same queue will be assigned back with
some capacity based on available capacity. If capacity is not there, queue will
be still 0. If many such queues with 0 capacity is starving, queue which got
submitted with an app first will be selected. We might need to consider
priority also here. {quote}
Discussed with [~wangda] and [~sunilg] offline. App priority is intra queue
and will not be usable as a criteria while considering to choose leaf queues
{quote} AbstractManagedParentQueue#validateQueueEntitlementChange is directly
operating on capacity. When absolute resource will get merged, this code will
be a problem? {quote}
This will require changes in policy class and could be addressed in a
separate jira once Aboslute resource is merged.
In initializeLimitsFromTemplate, should below code to be
setMaxApplications(leafQueueTemplate.getMaxApps());
related to parent queue as well. What if some one provided more apps in
template which could violate parent max-apps?
{quote} In validateConfigurations, does 0 a valid capacity? one could configure
0 as capacity and +ve integer for max-capacity?{quote}
Yes, 0 is a valid configuration for queue capacity
{quote}
How to configure orderingPolicy for AutoCreatedLeafQueue?
in below code, better to avoid _ for queue config names
{quote}
Can be configured through
<parent-queue-prefix>.leaf-queue-template.ordering-policy.
{quote}
1684 public static final String QUEUE_MANAGEMENT_MONITORING_INTERVAL =
1685 QUEUE_MANAGEMENT_CONFIG_PREFIX + "monitoring_interval";
{quote}
Fixed
{quote}
In CapacitySchedulerContext, better to use MonotonicClock
{quote}
Fixed
{quote}
In LeafQueue,
2011 public void setMaxAMResourcePerQueuePercent(
2012 float maxAMResourcePerQueuePercent) {
2013 this.maxAMResourcePerQueuePercent = maxAMResourcePerQueuePercent;
2014 }
how are we handling node labels?
{quote}
Discussed with [~wangda] and [~sunilg] offline. This has been addressed
through changes for using CapacitySchedulerConfiguration instead of individual
configs for a ManagedParentQueue
{quote}
PendingApplicationComparator could reuse existing fifo/fair app comparators?
{quote}
Discussed with [~wangda] and [~sunilg] offline. fifo/fair app comparators
cant be reused since they are mostly for intra queue app ordering and we cannot
consider priority etc while ordering leaf queues within a parent queue as
mentioned above.
Also this patch reverts changes done for using a common AutoCreatedLeafQueue
for both reservations and auto created queues. This was needed since the
initialize/reinitialize for ReservationQueues and AutoCreatedLeafQueue are no
longer common.
> Implement Framework and policy for capacity management of auto created queues
> ------------------------------------------------------------------------------
>
> Key: YARN-7473
> URL: https://issues.apache.org/jira/browse/YARN-7473
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Reporter: Suma Shivaprasad
> Assignee: Suma Shivaprasad
> Attachments: YARN-7473.1.patch, YARN-7473.10.patch,
> YARN-7473.11.patch, YARN-7473.2.patch, YARN-7473.3.patch, YARN-7473.4.patch,
> YARN-7473.5.patch, YARN-7473.6.patch, YARN-7473.7.patch, YARN-7473.8.patch,
> YARN-7473.9.patch
>
>
> This jira mainly addresses the following
>
> 1.Support adding pluggable policies on parent queue for dynamically managing
> capacity/state for leaf queues.
> 2. Implement a default policy that manages capacity based on pending
> applications and either grants guaranteed or zero capacity to queues based on
> parent's available guaranteed capacity.
> 3. Integrate with SchedulingEditPolicy framework to trigger this periodically
> and signal scheduler to take necessary actions for capacity/queue management.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
