[
https://issues.apache.org/jira/browse/YARN-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917986#comment-16917986
]
Eric Payne commented on YARN-9770:
----------------------------------
Hi [~jhung]. I'm sorry but I don't understand everything about the use case.
Can you please clarify the following:
bq. An underutilized queue which receives an application with many (thousands)
resource requests will hog scheduler allocations for a long time
Won't scheduling happen every time a (set of) node(s) heartbeats into the RM?
So, I don't understand what is blocking the scheduler for minutes. If the
default queue utilization ordering policy is being used, all queues should grow
at the same percentage, regardless of one large app in one queue and many
smaller apps in the other queues. What am I missing?
> Create a queue ordering policy which picks child queues with equal probability
> ------------------------------------------------------------------------------
>
> Key: YARN-9770
> URL: https://issues.apache.org/jira/browse/YARN-9770
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Jonathan Hung
> Assignee: Jonathan Hung
> Priority: Major
> Labels: release-blocker
> Attachments: YARN-9770.001.patch, YARN-9770.002.patch,
> YARN-9770.003.patch
>
>
> Ran some simulations with the default queue_utilization_ordering_policy:
> An underutilized queue which receives an application with many (thousands)
> resource requests will hog scheduler allocations for a long time (on the
> order of a minute). In the meantime apps are getting submitted to all other
> queues, which increases activeUsers in these queues, which drops user limit
> in these queues to small values if minimum-user-limit-percent is configured
> to small values (e.g. 10%).
> To avoid this issue, we assign to queues with equal probability, to avoid
> scenarios where queues don't get allocations for a long time.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
