On Wed, Jul 02, 2014 at 10:54:49AM -0700, Clint Byrum wrote: > Excerpts from Qiming Teng's message of 2014-07-02 00:02:14 -0700: > > Just some random thoughts below ... > > > > On Tue, Jul 01, 2014 at 03:47:03PM -0400, Mike Spreitzer wrote: > > > In AWS, an autoscaling group includes health maintenance functionality > > > --- > > > both an ability to detect basic forms of failures and an ability to react > > > properly to failures detected by itself or by a load balancer. What is > > > the thinking about how to get this functionality in OpenStack? Since > > > > We are prototyping a solution to this problem at IBM Research - China > > lab. The idea is to leverage oslo.messaging and ceilometer events for > > instance (possibly other resource such as port, securitygroup ...) > > failure detection and handling. > > > > Hm.. perhaps you should be contributing some reviews here as you may > have some real insight: > > https://review.openstack.org/#/c/100012/ > > This sounds a lot like what we're working on for continuous convergence.
Great. I will look into this specs and see if I can contribute some ideas. > > > OpenStack's OS::Heat::AutoScalingGroup has a more general member type, > > > what is the thinking about what failure detection means (and how it would > > > be accomplished, communicated)? > > > > When most OpenStack services are making use of oslo.notify, in theory, a > > service should be able to send/receive events related to resource > > status. In our current prototype, at least host failure (detected in > > Nova and reported with a patch), VM failure (detected by nova), and some > > lifecycle events of other resources can be detected and then collected > > by Ceilometer. There is certainly a possibility to listen to the > > message queue directly from Heat, but we only implemented the Ceilometer > > centric approach. > > > > > > > > I have not found design discussion of this; have I missed something? > > > > > > I suppose the natural answer for OpenStack would be centered around > > > webhooks. An OpenStack scaling group (OS SG = OS::Heat::AutoScalingGroup > > > or AWS::AutoScaling::AutoScalingGroup or OS::Heat::ResourceGroup or > > > OS::Heat::InstanceGroup) could generate a webhook per member, with the > > > meaning of the webhook being that the member has been detected as dead > > > and > > > should be deleted and removed from the group --- and a replacement member > > > created if needed to respect the group's minimum size. > > > > Well, I would suggest we generalize this into a event messaging or > > signaling solution, instead of just 'webhooks'. The reason is that > > webhooks as it is implemented today is not carrying a payload of useful > > information -- I'm referring to the alarms in Ceilometer. > > > > There are other cases as well. A member failure could be caused by a > > temporary communication problem, which means it may show up quickly when > > a replacement member is already being created. It may mean that we have > > to respond to an 'online' event in addition to an 'offline' event? > > > > The ideas behind convergence help a lot here. Skew happens in distributed > systems, so we expect it constantly. In the extra-capacity situation > above, we would just deal with it by scaling back down. There are also > situations where we might accidentally create two physical resources > because we got a 500 from the API but it was after the resource was > being created. This is the same problem, and has the same answer: pick > one and scale down (and if this is a critical server like a database, > we'll need lifecycle callbacks that will prevent suddenly killing a node > that would cost you uptime or recovery time). Glad to know this is considered and handled with a generic solution. As for recovering a server, I still suggest we have a per-resource-type restart logic. In the case of a nested stack, callbacks seem the right way to go. > > > When the member is > > > a Compute instance and Ceilometer exists, the OS SG could define a > > > Ceilometer alarm for each member (by including these alarms in the > > > template generated for the nested stack that is the SG), programmed to > > > hit > > > the member's deletion webhook when death is detected (I imagine there are > > > a few ways to write a Ceilometer condition that detects instance death). > > > > Yes. Compute instance failure can be detected with a Ceilometer plugin. > > In our prototype, we developed a Dispatcher plugin that can handle > > events like 'compute.instance.delete.end', 'compute.instance.create.end' > > after they have been processed based on a event_definitions.yaml file. > > There could be other ways, I think. > > > > The problem here today is about the recovery of SG member. If it is a > > compute instance, we can 'reboot', 'rebuild', 'evacuate', 'migrate' it, > > just to name a few options. The most brutal way to do this is like what > > HARestarter is doing today -- delete followed by a create. > > > > Right, so lifecycle callbacks are useful here, as we can expose an > interface for delaying and even cancelling a lifecycle event. > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev