Re: [Openstack] VM High Availability and Floating IP
Il giorno 26/lug/2012, alle ore 04:59, Angus Salkeld ha scritto: > On 24/07/12 12:12 -0700, Steven Dake wrote: >> On 07/24/2012 10:08 AM, Alessandro Tagliapietra wrote: >>> But i don't see any part (except the future plans) talking about HA at >>> instance level, that seems more to an application level >>> >> >> Currently heat's healthchecking is application only, but we do intend to >> improve instance healthchecking in v6. We are currently in our v5 >> development which concludes on July 30th. We have 4-5 week development >> windows. I'll make certain vm healthchecking is solid for v6 around >> August. However, we may be able to fit into v5 - I'll have to check >> with the main author of the HA feature (Angus Salkeld). > > I have just added a template demonstating an instance sending > healthchecks and getting restarted if it fails to send a healthcheck > within a specified period. > > https://github.com/heat-api/heat/blob/master/templates/WordPress_Single_Instance_With_IHA.template > > (just to give you another option). Thanks for that but I don't think that just restarting starts the vm on the other node if it fails. Best -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it > > -Angus > >> >> The HA feature HOWTO is described a bit here: >> >> https://github.com/heat-api/heat/wiki/Using-HA >> >> An example HA application template is here: >> >> https://github.com/heat-api/heat/blob/master/templates/WordPress_Single_Instance_With_HA.template >> >> This template launches a wordpress cloud application. If httpd or mysql >> fail, they are restarted. If they fail 3 times in 5 minutes, the entire >> VM is restarted (this is called escalation) under the assumption that >> the VM container is defective in some way. >> >> Regards >> -steve >> >>> Il giorno 24/lug/2012, alle ore 18:56, Jay Pipes ha scritto: >>> On 07/24/2012 12:52 PM, Alessandro Tagliapietra wrote: > Thank you Jay, never read about that. > Seems something like scalr/chef? WHich handles application and keeps a > minimum number of vm running? Yeah, kinda.. just one more way of doing things... :) -jay >>> >>> >>> ___ >>> Mailing list: https://launchpad.net/~openstack >>> Post to : openstack@lists.launchpad.net >>> Unsubscribe : https://launchpad.net/~openstack >>> More help : https://help.launchpad.net/ListHelp >>> >> >> >> >> ___ >> Mailing list: https://launchpad.net/~openstack >> Post to : openstack@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] VM High Availability and Floating IP
On 24/07/12 12:12 -0700, Steven Dake wrote: On 07/24/2012 10:08 AM, Alessandro Tagliapietra wrote: But i don't see any part (except the future plans) talking about HA at instance level, that seems more to an application level Currently heat's healthchecking is application only, but we do intend to improve instance healthchecking in v6. We are currently in our v5 development which concludes on July 30th. We have 4-5 week development windows. I'll make certain vm healthchecking is solid for v6 around August. However, we may be able to fit into v5 - I'll have to check with the main author of the HA feature (Angus Salkeld). I have just added a template demonstating an instance sending healthchecks and getting restarted if it fails to send a healthcheck within a specified period. https://github.com/heat-api/heat/blob/master/templates/WordPress_Single_Instance_With_IHA.template (just to give you another option). -Angus The HA feature HOWTO is described a bit here: https://github.com/heat-api/heat/wiki/Using-HA An example HA application template is here: https://github.com/heat-api/heat/blob/master/templates/WordPress_Single_Instance_With_HA.template This template launches a wordpress cloud application. If httpd or mysql fail, they are restarted. If they fail 3 times in 5 minutes, the entire VM is restarted (this is called escalation) under the assumption that the VM container is defective in some way. Regards -steve Il giorno 24/lug/2012, alle ore 18:56, Jay Pipes ha scritto: On 07/24/2012 12:52 PM, Alessandro Tagliapietra wrote: Thank you Jay, never read about that. Seems something like scalr/chef? WHich handles application and keeps a minimum number of vm running? Yeah, kinda.. just one more way of doing things... :) -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] VM High Availability and Floating IP
That's exactly what I wanted to achieve. Consider me as one of the testers if needed ;) Best Alessandro Il giorno 24/lug/2012, alle ore 20:24, Alex Glikson ha scritto: > We are working on a patch that would enable such capability within OpenStack > Nova. > It is a bit more complex than what you suggested. In a nutshell, one would > need to do the following: > - re-create the VM on the destination host > - re-attach volumes/IPs > - update DB > - make sure that if/when the original host recovers, it does not cause > corruption of the instance file on shared storage (e.g., by removing the > corresponding VM from the original host on startup) > > Regards, > Alex > > > > Alex Glikson > Manager, Cloud Operating System Technologies, IBM Haifa Research Lab > http://w3.haifa.ibm.com/dept/stt/cloud_sys.html | > https://www.research.ibm.com/haifa/dept/stt/cloud_sys.shtml > Email: glik...@il.ibm.com | Phone: +972-4-8281085 | Mobile: +972-54-647 | > Fax: +972-4-8296112 > > > > > From:Alessandro Tagliapietra > To:Jay Pipes , > Cc: openstack@lists.launchpad.net > Date: 24/07/2012 08:52 PM > Subject:Re: [Openstack] VM High Availability and Floating IP > Sent by:openstack-bounces+glikson=il.ibm@lists.launchpad.net > > > > But i don't see any part (except the future plans) talking about HA at > instance level, that seems more to an application level > > Il giorno 24/lug/2012, alle ore 18:56, Jay Pipes ha scritto: > > > On 07/24/2012 12:52 PM, Alessandro Tagliapietra wrote: > >> Thank you Jay, never read about that. > >> Seems something like scalr/chef? WHich handles application and keeps a > >> minimum number of vm running? > > > > Yeah, kinda.. just one more way of doing things... :) > > -jay > > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] VM High Availability and Floating IP
On 07/24/2012 09:52 AM, Alessandro Tagliapietra wrote: > Thank you Jay, never read about that. > Seems something like scalr/chef? WHich handles application and keeps a > minimum number of vm running? > The idea of keeping a minimum number of VMs running based upon VM load is called auto-scaling. We have added auto-scaling to our v5 release (which is targeted for July 30th). As far as puppet/chef integration goes, CloudFormation integrates well with both. For chef read: http://www.full360.com/blogs/integrating-aws-cloudformation-and-chef For puppet read: https://s3.amazonaws.com/cloudformation-examples/IntegratingAWSCloudFormationWithPuppet.pdf Note while these links talk about "AWS CloudFormation", Heat is essentially an AWS CloudFormation implementation for OpenStack. If you want to get started and give heat a try on Fedora 17+ or Ubuntu Precise, our getting started guides are in our wiki: https://github.com/heat-api/heat/wiki Let me know if you have follow-up questions. Regards -steve > Best > > Alessandro > > Il giorno 24/lug/2012, alle ore 14:34, Jay Pipes ha scritto: > >> On 07/24/2012 04:29 AM, Alessandro Tagliapietra wrote: >>> Hi guys, >>> >>> i've 2 missing pieces in my HA openstack install. Actually all openstack >>> services are managed by pacemaker and i can succesfully start/stop vm >>> etc. when the cloud controller is down (i've only 2 servers atm). >>> >>> 1 - how can i make a VM HA? Actually live-migration works fine, but if a >>> host goes down, how can i restart the vm on the other host? Should i >>> edit the 'host' column in the db and issue the restart of the vm? Any >>> other way? >> >> Check out that HEAT API: >> >> https://github.com/heat-api/heat/wiki/ >> >>> 2 - i've the servers hosted at Hetzner, for floating ip we've bought >>> failover ip which are assigned to each host and can be changed via the >>> api. So i have to make sure that if vm is on host1, floating ip >>> associated to the vm is routed to host1. My idea was to run a job that >>> checks the floating ip already associated to any vm, then queries the vm >>> info, checks on which host it's running and if it's different from the >>> other check, calls the hetzner api to switch the ip to the other server. >>> Any other idea? >> >> See above :) >> >> Best, >> -jay >> >>> Thanks in advance >>> >>> Best Regards >>> >>> -- >>> Alessandro Tagliapietra | VISup srl >>> piazza 4 novembre 7 >>> 20124 Milano >>> >>> http://www.visup.it >>> >>> >>> >>> ___ >>> Mailing list: https://launchpad.net/~openstack >>> Post to : openstack@lists.launchpad.net >>> Unsubscribe : https://launchpad.net/~openstack >>> More help : https://help.launchpad.net/ListHelp >>> >> >> ___ >> Mailing list: https://launchpad.net/~openstack >> Post to : openstack@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp > > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] VM High Availability and Floating IP
On 07/24/2012 10:08 AM, Alessandro Tagliapietra wrote: > But i don't see any part (except the future plans) talking about HA at > instance level, that seems more to an application level > Currently heat's healthchecking is application only, but we do intend to improve instance healthchecking in v6. We are currently in our v5 development which concludes on July 30th. We have 4-5 week development windows. I'll make certain vm healthchecking is solid for v6 around August. However, we may be able to fit into v5 - I'll have to check with the main author of the HA feature (Angus Salkeld). The HA feature HOWTO is described a bit here: https://github.com/heat-api/heat/wiki/Using-HA An example HA application template is here: https://github.com/heat-api/heat/blob/master/templates/WordPress_Single_Instance_With_HA.template This template launches a wordpress cloud application. If httpd or mysql fail, they are restarted. If they fail 3 times in 5 minutes, the entire VM is restarted (this is called escalation) under the assumption that the VM container is defective in some way. Regards -steve > Il giorno 24/lug/2012, alle ore 18:56, Jay Pipes ha scritto: > >> On 07/24/2012 12:52 PM, Alessandro Tagliapietra wrote: >>> Thank you Jay, never read about that. >>> Seems something like scalr/chef? WHich handles application and keeps a >>> minimum number of vm running? >> >> Yeah, kinda.. just one more way of doing things... :) >> -jay > > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] VM High Availability and Floating IP
We are working on a patch that would enable such capability within OpenStack Nova. It is a bit more complex than what you suggested. In a nutshell, one would need to do the following: - re-create the VM on the destination host - re-attach volumes/IPs - update DB - make sure that if/when the original host recovers, it does not cause corruption of the instance file on shared storage (e.g., by removing the corresponding VM from the original host on startup) Regards, Alex Alex Glikson Manager, Cloud Operating System Technologies, IBM Haifa Research Lab http://w3.haifa.ibm.com/dept/stt/cloud_sys.html | https://www.research.ibm.com/haifa/dept/stt/cloud_sys.shtml Email: glik...@il.ibm.com | Phone: +972-4-8281085 | Mobile: +972-54-647 | Fax: +972-4-8296112 From: Alessandro Tagliapietra To: Jay Pipes , Cc: openstack@lists.launchpad.net Date: 24/07/2012 08:52 PM Subject:Re: [Openstack] VM High Availability and Floating IP Sent by:openstack-bounces+glikson=il.ibm@lists.launchpad.net But i don't see any part (except the future plans) talking about HA at instance level, that seems more to an application level Il giorno 24/lug/2012, alle ore 18:56, Jay Pipes ha scritto: > On 07/24/2012 12:52 PM, Alessandro Tagliapietra wrote: >> Thank you Jay, never read about that. >> Seems something like scalr/chef? WHich handles application and keeps a minimum number of vm running? > > Yeah, kinda.. just one more way of doing things... :) > -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] VM High Availability and Floating IP
But i don't see any part (except the future plans) talking about HA at instance level, that seems more to an application level Il giorno 24/lug/2012, alle ore 18:56, Jay Pipes ha scritto: > On 07/24/2012 12:52 PM, Alessandro Tagliapietra wrote: >> Thank you Jay, never read about that. >> Seems something like scalr/chef? WHich handles application and keeps a >> minimum number of vm running? > > Yeah, kinda.. just one more way of doing things... :) > -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] VM High Availability and Floating IP
On 07/24/2012 12:52 PM, Alessandro Tagliapietra wrote: > Thank you Jay, never read about that. > Seems something like scalr/chef? WHich handles application and keeps a > minimum number of vm running? Yeah, kinda.. just one more way of doing things... :) -jay ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] VM High Availability and Floating IP
Thank you Jay, never read about that. Seems something like scalr/chef? WHich handles application and keeps a minimum number of vm running? Best Alessandro Il giorno 24/lug/2012, alle ore 14:34, Jay Pipes ha scritto: > On 07/24/2012 04:29 AM, Alessandro Tagliapietra wrote: >> Hi guys, >> >> i've 2 missing pieces in my HA openstack install. Actually all openstack >> services are managed by pacemaker and i can succesfully start/stop vm >> etc. when the cloud controller is down (i've only 2 servers atm). >> >> 1 - how can i make a VM HA? Actually live-migration works fine, but if a >> host goes down, how can i restart the vm on the other host? Should i >> edit the 'host' column in the db and issue the restart of the vm? Any >> other way? > > Check out that HEAT API: > > https://github.com/heat-api/heat/wiki/ > >> 2 - i've the servers hosted at Hetzner, for floating ip we've bought >> failover ip which are assigned to each host and can be changed via the >> api. So i have to make sure that if vm is on host1, floating ip >> associated to the vm is routed to host1. My idea was to run a job that >> checks the floating ip already associated to any vm, then queries the vm >> info, checks on which host it's running and if it's different from the >> other check, calls the hetzner api to switch the ip to the other server. >> Any other idea? > > See above :) > > Best, > -jay > >> Thanks in advance >> >> Best Regards >> >> -- >> Alessandro Tagliapietra | VISup srl >> piazza 4 novembre 7 >> 20124 Milano >> >> http://www.visup.it >> >> >> >> ___ >> Mailing list: https://launchpad.net/~openstack >> Post to : openstack@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~openstack >> More help : https://help.launchpad.net/ListHelp >> > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] VM High Availability and Floating IP
On 07/24/2012 04:29 AM, Alessandro Tagliapietra wrote: > Hi guys, > > i've 2 missing pieces in my HA openstack install. Actually all openstack > services are managed by pacemaker and i can succesfully start/stop vm > etc. when the cloud controller is down (i've only 2 servers atm). > > 1 - how can i make a VM HA? Actually live-migration works fine, but if a > host goes down, how can i restart the vm on the other host? Should i > edit the 'host' column in the db and issue the restart of the vm? Any > other way? Check out that HEAT API: https://github.com/heat-api/heat/wiki/ > 2 - i've the servers hosted at Hetzner, for floating ip we've bought > failover ip which are assigned to each host and can be changed via the > api. So i have to make sure that if vm is on host1, floating ip > associated to the vm is routed to host1. My idea was to run a job that > checks the floating ip already associated to any vm, then queries the vm > info, checks on which host it's running and if it's different from the > other check, calls the hetzner api to switch the ip to the other server. > Any other idea? See above :) Best, -jay > Thanks in advance > > Best Regards > > -- > Alessandro Tagliapietra | VISup srl > piazza 4 novembre 7 > 20124 Milano > > http://www.visup.it > > > > ___ > Mailing list: https://launchpad.net/~openstack > Post to : openstack@lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] VM High Availability and Floating IP
Hi guys, i've 2 missing pieces in my HA openstack install. Actually all openstack services are managed by pacemaker and i can succesfully start/stop vm etc. when the cloud controller is down (i've only 2 servers atm). 1 - how can i make a VM HA? Actually live-migration works fine, but if a host goes down, how can i restart the vm on the other host? Should i edit the 'host' column in the db and issue the restart of the vm? Any other way? 2 - i've the servers hosted at Hetzner, for floating ip we've bought failover ip which are assigned to each host and can be changed via the api. So i have to make sure that if vm is on host1, floating ip associated to the vm is routed to host1. My idea was to run a job that checks the floating ip already associated to any vm, then queries the vm info, checks on which host it's running and if it's different from the other check, calls the hetzner api to switch the ip to the other server. Any other idea? Thanks in advance Best Regards -- Alessandro Tagliapietra | VISup srl piazza 4 novembre 7 20124 Milano http://www.visup.it ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
