[ https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17247614#comment-17247614 ]
Wangda Tan commented on YARN-10504: ----------------------------------- Thanks [~zhuqi], as we discussed, we're working on the patch in parallel, it will be great if you can let us know if you have intention to work on a patch so we can coordinate in the future. Also, as I shared offline to you, the patch you attached has the following issues: 1) It changes CS config object, which makes hard for us to identify if it is a change caused by config file change or internal change. 2) Auto created queue (which is why we add this feature) doesn't load config from CS config. I attached patch (see ver.2). It takes the following approach: 1) Major updates are in updateClusterResource, which will be called after scheduler init and reinit. 2) Introduced new fields to QueueCapacities (weight, and normalized_weight). Based on normalized_weight, we will calculate absolute resource. 3) Did a bunch of refactorings because code to handle changes of queue efficient resource and config is everywhere, very hard to troubleshoot and make further changes. I hope the refactoring in the patch can help code easier maintained. Can you please help to review it? [~zhuqi], [~bteke], [~pbacsko], [~shuzirra], [~snemeth], [~sunilg] > Implement weight mode in Capacity Scheduler > ------------------------------------------- > > Key: YARN-10504 > URL: https://issues.apache.org/jira/browse/YARN-10504 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Benjamin Teke > Assignee: zhuqi > Priority: Major > Attachments: YARN-10504.001.patch, YARN-10504.ver-1.patch, > YARN-10504.ver-2.patch > > > To allow the possibility to flexibly create queues in Capacity Scheduler a > weight mode should be introduced. The existing \{{capacity }}property should > be used with a different syntax, i.e: > root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0 > root.users.capacity = 1.0w > root.users.capacity = w:1.0 > Weight support should not impact the existing functionality. > > The new functionality should: > * accept and validate the new weight values > * enforce a singular mode on the whole queue tree > * (re)calculate the relative (percentage-based) capacities based on the > weights during launch and every time the queue structure changes -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org