Re: [Openstack-operators] [nova] VM HA support in trunk
Hi Kazu, thanks for this update. Sorry I am a bit late in replying to this thread, but one of my students just ran into an issue running pacemaker-based evacuation of hosts. It seems that pacemaker 1.1.10 is not supposed to work with remote, and the 14.04 distro comes with that version. Did you get remote to work, if so how? The pull request [1] indicates that remote support was added, but its unclear how the above version difference was handled. Did you people resort to compiling the latest pcm from source or something else? Affan [1] https://github.com/ntt-sic/masakari/pull/11 On Fri, 19 Feb 2016 at 09:19 Toshikazu Ichikawa < ichikawa.toshik...@lab.ntt.co.jp> wrote: > Hi Affan, > > > > Pacemaker works fine on either a canonical distribution or RDO. > > I use our tool [1] using Pacemaker on Ubuntu without any specific issue. > > > > [1] https://github.com/ntt-sic/masakari > > > > Thanks, > > Kazu > > > > *From:* Affan Syed [mailto:affan.syed@gmail.com] > *Sent:* Tuesday, February 16, 2016 2:02 PM > *To:* Matt Fischer <m...@mattfischer.com>; Toshikazu Ichikawa < > ichikawa.toshik...@lab.ntt.co.jp> > *Cc:* openstack-operators@lists.openstack.org > *Subject:* Re: [Openstack-operators] [nova] VM HA support in trunk > > > > Hi Kazu and Matt, > > Thanks for the pointers. I think the discussion around pacemaker and > pacemaker remote seems most promising, esp with Russel's blog post I found > after I emailed earlier [1]. > > > > Not sure how tooling would be different, but pacemaker, given its use in > the controller cluster anyways, seems a more logical choice. Any issues you > people think with a canonical distribution instead of RDO? > > > > Affan > > > > > > [1] > http://blog.russellbryant.net/2015/03/10/the-different-facets-of-openstack-ha/ > > > > On Mon, 15 Feb 2016 at 20:59 Matt Fischer <m...@mattfischer.com> wrote: > > I believe that either have your customers design their apps to handle > failures or have tools that are reactive to failures. > > > > Unfortunately like many other private cloud operators we deal a lot with > legacy applications that aren't scaled horizontally or fault tolerant and > so we've built tooling to handle customer notifications (reactive). When we > lose a compute host we generate a notice to customers and then work on > evacuating their instances. For the evac portion nova host-evacuate or > host-evacuate-live work fairly well, although we rarely get a functioning > floating-IP after host-evacuate without other work. > > > > Getting adoption of heat or other automation tooling to educate customers > is a long process, especially when they're used to VMware where I think > they get the VM HA stuff for "free". > > > > > > On Mon, Feb 15, 2016 at 8:25 AM, Toshikazu Ichikawa < > ichikawa.toshik...@lab.ntt.co.jp> wrote: > > Hi Affan, > > > > > > I don’t think any components in Liberty provide HA VM support directly. > > > > However, many works are published and open-sourced, here. > > https://etherpad.openstack.org/p/automatic-evacuation > > You may find ideas and solutions. > > > > And, the discussion on this topic is on-going at HA meeting. > > https://wiki.openstack.org/wiki/Meetings/HATeamMeeting > > > > thanks, > > Kazu > > > > *From:* Affan Syed [mailto:affan.syed@gmail.com] > *Sent:* Monday, February 15, 2016 12:51 PM > *To:* openstack-operators@lists.openstack.org > *Subject:* [Openstack-operators] [nova] VM HA support in trunk > > > > reposting with the correct tag, hopefully. Would really appreciate some > pointers. > > -- Forwarded message - > From: Affan Syed <affan.syed@gmail.com> > Date: Sat, 13 Feb 2016 at 15:13 > Subject: [nova] VM HA support in trunk > To: <openstack-operators@lists.openstack.org> > > > > Hi all, > > I have been trying to understand if we currently have some VM HA support > as part of Liberty? > > > > To be precise, how are host being down due to power failure handled, > specifically in terms of migrating the VMs but possibly even their > networking configs (tunnels etc). > > > > The VM migration like XEN-HA or KVM cluster seem to require 1+1 HA, I have > read a few places about celiometer+heat templates to launch VMs for an N+1 > backup scenario, but these all seem like one-off setups. > > > > > > This issue seems to be very much important for legacy enterprises to move > their "pets" --- not sure if we can simply wish away that mindset! > > > > Affan > > > > > > > > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [nova] VM HA support in trunk
Matt/Joe, I think your points are valid. However, when looking at woowing customers who are in legacy operation, doing all the changes at once doesnt seem like a viable value proposition. This first order transition is important to get them to see the benefits of cloud. Then we can have their previous OPS people to spend spare time on becoming DEVss and build cloud native apps ! Affan On Tue, 16 Feb 2016 at 19:24 Bajin, Joseph <jba...@verisign.com> wrote: > I would have to agree with Matt. The ability for any sort of handling of > failures either reside within the application or tools around the > application to make it work. Having the infrastructure handle the > failures, I believe, is a slippery slope that is starting to appear more > and more. > > I do fear that many people/organizations are starting to look at the cloud > as a “low cost” or “free” VMWare solution. They want the same enterprise > based availability and support that they get with a vendor paid solution > without the cost of the vendor paid solution. I have started to see and > hear more about how vendors are adding “enterprise” solutions to > OpenStack. This includes High Availability features that rely on the > infrastructure to manage instead of the application. I fear the direction > of all the projects will begin migrating this way as more vendors get > involve and want to figure out business models that they can use around > “enterprise” feature-sets. > > —Joe > > > > From: Matt Fischer <m...@mattfischer.com> > Date: Monday, February 15, 2016 at 10:59 AM > To: Toshikazu Ichikawa <ichikawa.toshik...@lab.ntt.co.jp> > Cc: "openstack-operators@lists.openstack.org" < > openstack-operators@lists.openstack.org> > Subject: Re: [Openstack-operators] [nova] VM HA support in trunk > > I believe that either have your customers design their apps to handle > failures or have tools that are reactive to failures. > > Unfortunately like many other private cloud operators we deal a lot with > legacy applications that aren't scaled horizontally or fault tolerant and > so we've built tooling to handle customer notifications (reactive). When we > lose a compute host we generate a notice to customers and then work on > evacuating their instances. For the evac portion nova host-evacuate or > host-evacuate-live work fairly well, although we rarely get a functioning > floating-IP after host-evacuate without other work. > > Getting adoption of heat or other automation tooling to educate customers > is a long process, especially when they're used to VMware where I think > they get the VM HA stuff for "free". > > > On Mon, Feb 15, 2016 at 8:25 AM, Toshikazu Ichikawa < > ichikawa.toshik...@lab.ntt.co.jp> wrote: > >> Hi Affan, >> >> >> >> >> >> I don’t think any components in Liberty provide HA VM support directly. >> >> >> >> However, many works are published and open-sourced, here. >> >> https://etherpad.openstack.org/p/automatic-evacuation >> >> You may find ideas and solutions. >> >> >> >> And, the discussion on this topic is on-going at HA meeting. >> >> https://wiki.openstack.org/wiki/Meetings/HATeamMeeting >> >> >> >> thanks, >> >> Kazu >> >> >> >> *From:* Affan Syed [mailto:affan.syed@gmail.com] >> *Sent:* Monday, February 15, 2016 12:51 PM >> *To:* openstack-operators@lists.openstack.org >> *Subject:* [Openstack-operators] [nova] VM HA support in trunk >> >> >> >> reposting with the correct tag, hopefully. Would really appreciate some >> pointers. >> >> -- Forwarded message - >> From: Affan Syed <affan.syed@gmail.com> >> Date: Sat, 13 Feb 2016 at 15:13 >> Subject: [nova] VM HA support in trunk >> To: <openstack-operators@lists.openstack.org> >> >> >> >> Hi all, >> >> I have been trying to understand if we currently have some VM HA support >> as part of Liberty? >> >> >> >> To be precise, how are host being down due to power failure handled, >> specifically in terms of migrating the VMs but possibly even their >> networking configs (tunnels etc). >> >> >> >> The VM migration like XEN-HA or KVM cluster seem to require 1+1 HA, I >> have read a few places about celiometer+heat templates to launch VMs for an >> N+1 backup scenario, but these all seem like one-off setups. >> >> >> >> >> >> This issue seems to be very much important for legacy enterprises to move >> their "pets" --- not sure if w
Re: [Openstack-operators] [nova] VM HA support in trunk
Hi Kazu and Matt, Thanks for the pointers. I think the discussion around pacemaker and pacemaker remote seems most promising, esp with Russel's blog post I found after I emailed earlier [1]. Not sure how tooling would be different, but pacemaker, given its use in the controller cluster anyways, seems a more logical choice. Any issues you people think with a canonical distribution instead of RDO? Affan [1] http://blog.russellbryant.net/2015/03/10/the-different-facets-of-openstack-ha/ On Mon, 15 Feb 2016 at 20:59 Matt Fischer <m...@mattfischer.com> wrote: > I believe that either have your customers design their apps to handle > failures or have tools that are reactive to failures. > > Unfortunately like many other private cloud operators we deal a lot with > legacy applications that aren't scaled horizontally or fault tolerant and > so we've built tooling to handle customer notifications (reactive). When we > lose a compute host we generate a notice to customers and then work on > evacuating their instances. For the evac portion nova host-evacuate or > host-evacuate-live work fairly well, although we rarely get a functioning > floating-IP after host-evacuate without other work. > > Getting adoption of heat or other automation tooling to educate customers > is a long process, especially when they're used to VMware where I think > they get the VM HA stuff for "free". > > > On Mon, Feb 15, 2016 at 8:25 AM, Toshikazu Ichikawa < > ichikawa.toshik...@lab.ntt.co.jp> wrote: > >> Hi Affan, >> >> >> >> >> >> I don’t think any components in Liberty provide HA VM support directly. >> >> >> >> However, many works are published and open-sourced, here. >> >> https://etherpad.openstack.org/p/automatic-evacuation >> >> You may find ideas and solutions. >> >> >> >> And, the discussion on this topic is on-going at HA meeting. >> >> https://wiki.openstack.org/wiki/Meetings/HATeamMeeting >> >> >> >> thanks, >> >> Kazu >> >> >> >> *From:* Affan Syed [mailto:affan.syed@gmail.com] >> *Sent:* Monday, February 15, 2016 12:51 PM >> *To:* openstack-operators@lists.openstack.org >> *Subject:* [Openstack-operators] [nova] VM HA support in trunk >> >> >> >> reposting with the correct tag, hopefully. Would really appreciate some >> pointers. >> >> -- Forwarded message - >> From: Affan Syed <affan.syed@gmail.com> >> Date: Sat, 13 Feb 2016 at 15:13 >> Subject: [nova] VM HA support in trunk >> To: <openstack-operators@lists.openstack.org> >> >> >> >> Hi all, >> >> I have been trying to understand if we currently have some VM HA support >> as part of Liberty? >> >> >> >> To be precise, how are host being down due to power failure handled, >> specifically in terms of migrating the VMs but possibly even their >> networking configs (tunnels etc). >> >> >> >> The VM migration like XEN-HA or KVM cluster seem to require 1+1 HA, I >> have read a few places about celiometer+heat templates to launch VMs for an >> N+1 backup scenario, but these all seem like one-off setups. >> >> >> >> >> >> This issue seems to be very much important for legacy enterprises to move >> their "pets" --- not sure if we can simply wish away that mindset! >> >> >> >> Affan >> >> >> >> >> >> ___ >> OpenStack-operators mailing list >> OpenStack-operators@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> >> ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] [nova] VM HA support in trunk
Hi all, I have been trying to understand if we currently have some VM HA support as part of Liberty? To be precise, how are host being down due to power failure handled, specifically in terms of migrating the VMs but possibly even their networking configs (tunnels etc). The VM migration like XEN-HA or KVM cluster seem to require 1+1 HA, I have read a few places about celiometer+heat templates to launch VMs for an N+1 backup scenario, but these all seem like one-off setups. This issue seems to be very much important for legacy enterprises to move their "pets" --- not sure if we can simply wish away that mindset! Affan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
