Hi, On Wed, 2013-11-27 at 14:45 +0000, Edward Hope-Morley wrote: > Moving this to the ml as requested, would appreciate > comments/thoughts/feedback.
Thanks, I too would appreciate input from others. > So, I recently proposed a small patch to the oslo rpc code (initially in > oslo-incubator then moved to oslo.messaging) which extends the existing > support for limiting the rpc thread pool so that concurrent requests can > be limited based on type/method. The blueprint and patch are here: > > https://blueprints.launchpad.net/oslo.messaging/+spec/rpc-concurrency-control > > The basic idea is that if you have server with limited resources you may > want restrict operations that would impact those resources e.g. live > migrations on a specific hypervisor or volume formatting on particular > volume node. This patch allows you, admittedly in a very crude way, to > apply a fixed limit to a set of rpc methods. I would like to know > whether or not people think this is sort of thing would be useful or > whether it alludes to a more fundamental issue that should be dealt with > in a different manner. Just to be clear for everyone what we're talking about. Your patch means that if an operator sees that requests to the 'foo' and 'bar' RPC methods for a given service are overwhelming the capacity of the machine, you can throttle them by adding e.g. concurrency_control_enabled = true concurrency_control_actions = foo,bar concurrency_control_limit = 2 to the service's configuration file. If you accept the premise of what's required here, I think you really want to have e.g. a json policy file which can control the concurrency limit on each method individually: { "compute": { "baseapi": { "ping": 10 }, "": { "foo": 1, "bar": 2 } } } but that starts feeling pretty ridiculous. My concern is that we're avoiding addressing a more fundamental issue here. From IRC: <markmc> "avoid specific concurrent operations from consuming too many system resources and starving other less resource intensive actions" <markmc> I'd like us to think about whether we can come up with a solution that fixes the problem for people, without them having to mess with this type of configuration <markmc> but yeah ... if we can't figure out a way of doing that, there is an argument for giving operators and interim workaround <markmc> I wouldn't be in favour of an interim fix without first exploring the options for a more fundamental fix <markmc> this isn't easily removable later, because once people start to rely on it we would need to put it through a deprecation period to remove it <markmc> also, an interim solution like this takes away the pressure on us to find a more fundamental solution ... and we may wind up never doing that So, I guess my first question is ... what specific RPC methods have you seen issues with and feel you need to throttle? Thanks, Mark. _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev