[
https://issues.apache.org/jira/browse/YARN-8191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487573#comment-16487573
]
Gergo Repas commented on YARN-8191:
-----------------------------------
Thanks [~haibochen] for the review!
{quote}
In this patch, getRemovedStaticQueues() and QueueManager.setQueuesToDynamic()
together identify the queues that were added in the previous allocation file,
but are now removed in the new allocation file, and then mark them as dynamic.
What I mean is that, is it possible that some queues that were created
dynamically, but are now included in the new allocation file? If so, we need to
mark them as static.
{quote}
This is done in {{QueueManager.ensureQueueExistsAndIsCompatibleAndIsStatic()}}
method, it's been taken care of by the {{queue.setDynamic(false);}} line.
{quote}
The behavior of QueueManager.removeLeafQueue() is still changed with the
refactoring. Previously it would return true if there is no incompatible queue
found, but it now returns false. We should also return true if
removeEmptyIncompatibleQueues(name, FSQueueType.PARENT) returns null.
Similarly, in IncompatibleQueueRemovalTask.execute(), the task shall be removed
if `removed == null`.
{quote}
Thanks, I've corrected these conditions.
{quote}
Let's add some javadoc to newly added QueueManager public methods.
{quote}
Sure, I added them.
{quote}
`reloadListener.onCheck();` makes me worried what if the listener is not set.
Looking closely at the code, the setReloadListener() is always set right after
the AllocationFileLoaderService constructor, so I think we can move
reloadListener as a construnctor argument, so that we never worry if listener
is null.{quote}
The Listener is now a constructor argument in the latest patch.
> Fair scheduler: queue deletion without RM restart
> -------------------------------------------------
>
> Key: YARN-8191
> URL: https://issues.apache.org/jira/browse/YARN-8191
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
> Affects Versions: 3.0.1
> Reporter: Gergo Repas
> Assignee: Gergo Repas
> Priority: Major
> Attachments: Queue Deletion in Fair Scheduler.pdf,
> YARN-8191.000.patch, YARN-8191.001.patch, YARN-8191.002.patch,
> YARN-8191.003.patch, YARN-8191.004.patch, YARN-8191.005.patch,
> YARN-8191.006.patch, YARN-8191.007.patch, YARN-8191.008.patch,
> YARN-8191.009.patch, YARN-8191.010.patch, YARN-8191.011.patch,
> YARN-8191.012.patch, YARN-8191.013.patch, YARN-8191.014.patch,
> YARN-8191.015.patch
>
>
> The Fair Scheduler never cleans up queues even if they are deleted in the
> allocation file, or were dynamically created and are never going to be used
> again. Queues always remain in memory which leads to two following issues.
> # Steady fairshares aren’t calculated correctly due to remaining queues
> # WebUI shows deleted queues, which is confusing for users (YARN-4022).
> We want to support proper queue deletion without restarting the Resource
> Manager:
> # Static queues without any entries that are removed from fair-scheduler.xml
> should be deleted from memory.
> # Dynamic queues without any entries should be deleted.
> # RM Web UI should only show the queues defined in the scheduler at that
> point in time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
