[jira] [Commented] (YARN-6840) Leverage RMStateStore to store scheduler configuration updates
[ https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110380#comment-16110380 ] Zhe Zhang commented on YARN-6840: - Could this task leverage logic from HDFS-10631 and YARN-6900? > Leverage RMStateStore to store scheduler configuration updates > -- > > Key: YARN-6840 > URL: https://issues.apache.org/jira/browse/YARN-6840 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan > > With this change, user doesn't have to setup separate storage system (like > LevelDB) to store updates of scheduler configs. And dynamic queue can be used > when RM HA is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6840) Leverage RMStateStore to store scheduler configuration updates
[ https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16106562#comment-16106562 ] Esther Zheng commented on YARN-6840: test test test > Leverage RMStateStore to store scheduler configuration updates > -- > > Key: YARN-6840 > URL: https://issues.apache.org/jira/browse/YARN-6840 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan > > With this change, user doesn't have to setup separate storage system (like > LevelDB) to store updates of scheduler configs. And dynamic queue can be used > when RM HA is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6840) Leverage RMStateStore to store scheduler configuration updates
[ https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096913#comment-16096913 ] Wangda Tan commented on YARN-6840: -- [~Naganarasimha], thanks, I think compress could still help here: for serialized stream of LogMutation, we can compress it before send to backend store. > Leverage RMStateStore to store scheduler configuration updates > -- > > Key: YARN-6840 > URL: https://issues.apache.org/jira/browse/YARN-6840 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan > > With this change, user doesn't have to setup separate storage system (like > LevelDB) to store updates of scheduler configs. And dynamic queue can be used > when RM HA is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6840) Leverage RMStateStore to store scheduler configuration updates
[ https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096632#comment-16096632 ] Naganarasimha G R commented on YARN-6840: - [~wangda], Can help to check that out, IIUC Based on YARN-5947 LogMutations are the objects which are stored in zk right ? Further we need to consider the case of partitions also as the whole queue hierarchy mapping gets repeated for each partition. But as you said there would be decrease in size initially as the contents would get repeated but if we plan to store mutations/updates only, then compression might not be of any help for mutations ! > Leverage RMStateStore to store scheduler configuration updates > -- > > Key: YARN-6840 > URL: https://issues.apache.org/jira/browse/YARN-6840 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan > > With this change, user doesn't have to setup separate storage system (like > LevelDB) to store updates of scheduler configs. And dynamic queue can be used > when RM HA is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6840) Leverage RMStateStore to store scheduler configuration updates
[ https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095022#comment-16095022 ] Wangda Tan commented on YARN-6840: -- [~Naganarasimha], I think storing to a single zk node should be fine. Not sure if you could help to do an experiment: what is the size of the capacity-scheduler.xml with several hundreds queues after compression? I expect it should be less than 30kb. And as number of queue grows, compression rate should increase as well (more duplicated fields). We can also store diff of changes instead of store whole config file for every update. > Leverage RMStateStore to store scheduler configuration updates > -- > > Key: YARN-6840 > URL: https://issues.apache.org/jira/browse/YARN-6840 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan > > With this change, user doesn't have to setup separate storage system (like > LevelDB) to store updates of scheduler configs. And dynamic queue can be used > when RM HA is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6840) Leverage RMStateStore to store scheduler configuration updates
[ https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094007#comment-16094007 ] Naganarasimha G R commented on YARN-6840: - [~wangda], bq. Diff of config update should be small, and we can compress it before store, I expect less than 10KBs for each update for very bad cases (after compression). So your estimation of 10kb is for each queue update? In between do we plan to store the hierarchy as zookeeper nodes or whole queue configuration under a single node. In either case, we generally tend to see 400 ~ 500 queues with our customers and expect much more to grow, would it be of any concern ? > Leverage RMStateStore to store scheduler configuration updates > -- > > Key: YARN-6840 > URL: https://issues.apache.org/jira/browse/YARN-6840 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan > > With this change, user doesn't have to setup separate storage system (like > LevelDB) to store updates of scheduler configs. And dynamic queue can be used > when RM HA is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6840) Leverage RMStateStore to store scheduler configuration updates
[ https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093827#comment-16093827 ] Wangda Tan commented on YARN-6840: -- User has to make a choice: either using queue-refresh (reloading XML file) or using the the store-based config and use API to update configurations. YARN-6322 is tracking the change. > Leverage RMStateStore to store scheduler configuration updates > -- > > Key: YARN-6840 > URL: https://issues.apache.org/jira/browse/YARN-6840 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan > > With this change, user doesn't have to setup separate storage system (like > LevelDB) to store updates of scheduler configs. And dynamic queue can be used > when RM HA is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6840) Leverage RMStateStore to store scheduler configuration updates
[ https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093740#comment-16093740 ] Daniel Templeton commented on YARN-6840: Just curious, how do you plan to deal with updates to the scheduler config that happen from reloading the XML file? Or are you planning to disable reloading of the file? > Leverage RMStateStore to store scheduler configuration updates > -- > > Key: YARN-6840 > URL: https://issues.apache.org/jira/browse/YARN-6840 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan > > With this change, user doesn't have to setup separate storage system (like > LevelDB) to store updates of scheduler configs. And dynamic queue can be used > when RM HA is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6840) Leverage RMStateStore to store scheduler configuration updates
[ https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092636#comment-16092636 ] Wangda Tan commented on YARN-6840: -- [~dan...@cloudera.com], thanks for your comments. To me, scheduler configuration updates are not general data store. Diff of config update should be small, and we can compress it before store, I expect less than 10KBs for each update for very bad cases (after compression). And frequency of config update will be much slower than application information. So if we continue store states of application to RMStateStore, I didn't see any issue of store configs to RMStateStore. Please let me know if you think different. > Leverage RMStateStore to store scheduler configuration updates > -- > > Key: YARN-6840 > URL: https://issues.apache.org/jira/browse/YARN-6840 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan > > With this change, user doesn't have to setup separate storage system (like > LevelDB) to store updates of scheduler configs. And dynamic queue can be used > when RM HA is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6840) Leverage RMStateStore to store scheduler configuration updates
[ https://issues.apache.org/jira/browse/YARN-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092405#comment-16092405 ] Daniel Templeton commented on YARN-6840: Are you sure that's a good idea? We're already abusing ZK pretty badly with the amount of junk we cram in there as is. Zookeeper is pretty clear in the docs that it should not be used as a general purpose data store. > Leverage RMStateStore to store scheduler configuration updates > -- > > Key: YARN-6840 > URL: https://issues.apache.org/jira/browse/YARN-6840 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan > > With this change, user doesn't have to setup separate storage system (like > LevelDB) to store updates of scheduler configs. And dynamic queue can be used > when RM HA is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
