[
https://issues.apache.org/jira/browse/YARN-7004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Yang updated YARN-7004:
---------------------------
Summary: Add configs cache to optimize refreshQueues performance for large
scale of queues (was: Add configs cache to optimize refreshQueues performance
for large scale queues)
> Add configs cache to optimize refreshQueues performance for large scale of
> queues
> ---------------------------------------------------------------------------------
>
> Key: YARN-7004
> URL: https://issues.apache.org/jira/browse/YARN-7004
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
> Affects Versions: 2.9.0, 3.0.0-alpha4
> Reporter: Tao Yang
> Assignee: Tao Yang
> Attachments: YARN-7004.001.patch
>
>
> We have requirements for large scale queues in our production environment to
> serve for many projects. So we did some tests for more than 5000 queues and
> found that refreshQueues process took more than 1 minute. The refreshQueues
> process costs most of time on iterating over all configurations to get
> accessible-node-labels and ordering-policy configs for every queue.
> Loading queue configs from cache should be beneficial to reduce time costs
> (optimized from 1 minutes to 3 seconds for 5000 queues in our test) when
> initializing/reinitializing queues. So I propose to load queue configs into
> cache in CapacityScheduler#initializeQueues and
> CapacityScheduler#reinitializeQueues. If cache has not be loaded on other
> scenes, such as in test cases, it still can get queue configs by iterating
> over all configurations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
