[Openstack] Dashboard shows Keystone and Glance as localhost but they are not
Guys, My dashboard shows keystone and glance as localhost (127.0.0.1) but they are not locally installed and therefor showing offline. I have checked and double checked every config I could find and all are pointing to the actual location where keynote and glance are installed. (192.168.0.8) Here are my configs: keystone.conf: http://pastebin.com/NcgczLCd glance-registry.conf: http://pastebin.com/aRKGzMRb glance-api.conf: http://pastebin.com/gZMQHsQg glance-scrubber.conf: http://pastebin.com/HGCdMPz4 nova.conf: http://pastebin.com/2jc9aqDu local_settings.py (/etc/openstack-dashboard/ /usr/share/openstack-dashboard/local/) : http://pastebin.com/iPwbxeab What am I missing? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies.___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
Alright, picture is getting more clear. Thanks all! Got some extra questions tho: Font-end API servers -- what does this do? * load balanced with h/w load balancer * use s/w LB for smaller deployments * run nova-scheduler on each MySQL DB * multi-master configuration with pacemaker providing an floating IP. - seems to work. RabbitMQ service * I think there are some corner cases where messages could be dropped during failover and I believe later RabbitMQ versions support full multi-master but require some client side changes - is there any plans to support this? -- I would like to know this as well! nova-volume service * I think this can be solved by 'building' a proper cloud app which does not fail if one node fails where it is living on as it should be living on multiple nodes/sites. Glance * Run on multiple servers -- how do they get synced? Compute servers * Each run their own copy of nova-api (only instances running on the node use this) * nova-network (multi-host configuration) with private network -- Needs its own discussion I guess. Keystone * How about keystone, same as glance I guess? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Dan Wendlandt [mailto:d...@nicira.com] To: Ghe Rivero [mailto:ghe.riv...@stackops.com] Cc: netst...@lists.launchpad.net, darren.birk...@gmail.com, openstack@lists.launchpad.net Sent: Tue, 21 Feb 2012 18:13:55 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? 2012/2/21 Ghe Rivero ghe.riv...@stackops.com BigSwitch and Nicira have created different blueprints in quantum to add their plugins for essex-4 about ten days ago, Cisco already have it included almost from the beginning. No news of them for the moment, but i really hope that both of them make it in time! Ghe Rivero https://blueprints.launchpad.net/quantum/+spec/quantum-nvp-plugin https://blueprints.launchpad.net/quantum/+spec/quantum-bigswitch-plugin To be clear, there are several more existing and planned plugins for Quantum, even beyond the three you mention. Each plugin will make its own trade-offs in terms of HA for control plane and data plane components. I'm planning on organizing a talk at the Folsom summit/conference where people have a chance to talk about their plugins. No need for a flame war here :) One other point of confusion that I'd like to address is that there is NO requirement that a plugin be merged into the main Quantum repo for it to be a valid plugin. I know of several plugins that are used despite not being merged in (in fact, at the summit, we're going to discuss if ANY plugins should remain in the main repo... this is likely a larger decision that should be made consistently across all openstack projects). Dan 2012/2/21 Diego Parrilla Santamaría diego.parrilla.santama...@gmail.com This is the kind of flame war I love... ;-) if you think the HA challenges in Openstack Nova are at MySQL and Rabbit level, you are not putting the focus on the right components. The biggest HA (and scalability) challenges in Nova are in the networking side. There are a lot of companies tackling the networking challenges of Nova (BigSwitch, Nicira, Cisco, etc...), and I would like to see how they evolve in the future and how they will address this problem. Right now, only promises. Fight! -- Diego Parrilla CEO www.stackops.com | diego.parri...@stackops.com | +34 649 94 43 29 | skype:diegoparrilla On Tue, Feb 21, 2012 at 1:04 PM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Hi Darren, No, I am not looking at a performance increasment at this moment. Just trying to make the different components Nova is build on redundant. Of course a master/master setup of mysql / rabbitmq with a floating IP will probably work but it bugs me that Nova is not really build with flexibility of it components in mind. I understand it is impossible to cover every scenario and you can make things redundant into the infinitive but why doesn't have Nova any of this build in? Why do I need to setup a Virtual IP/floating IP where Nova simple could accept
Re: [Openstack] Dashboard shows Keystone and Glance as localhost but they are not
Yes, just found out with the help of lxu and zunzel: root@server10332:~# keystone-manage endpointTemplates list All EndpointTemplates service region Public URL --- novaRegionOne http://127.0.0.1:8774/v1.1/%tenant_id% glance RegionOne http://127.0.0.1:9292/v1 keystoneRegionOne http://127.0.0.1:5000/v2.0 Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Andiabes [mailto:andi.a...@gmail.com] To: i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl] Cc: openstack@lists.launchpad.net [mailto:openstack@lists.launchpad.net] Sent: Mon, 27 Feb 2012 13:58:16 +0100 Subject: Re: [Openstack] Dashboard shows Keystone and Glance as localhost but they are not Might be worth checking how those services are registered in keystone.. On Feb 27, 2012, at 7:06 AM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Guys, My dashboard shows keystone and glance as localhost (127.0.0.1) but they are not locally installed and therefor showing offline. I have checked and double checked every config I could find and all are pointing to the actual location where keynote and glance are installed. (192.168.0.8) Here are my configs: keystone.conf: http://pastebin.com/NcgczLCd glance-registry.conf: http://pastebin.com/aRKGzMRb glance-api.conf: http://pastebin.com/gZMQHsQg glance-scrubber.conf: http://pastebin.com/HGCdMPz4 nova.conf: http://pastebin.com/2jc9aqDu local_settings.py (/etc/openstack-dashboard/ /usr/share/openstack-dashboard/local/) : http://pastebin.com/iPwbxeab What am I missing? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
Ho Florian, I respect your opinion of course and I am not here to start a fight. Point is, how I understand it Nova really isn't very flexible on High Availability terms. I have now a master/master mysql setup with a Virtual IP working just to work around the shortcomings Nova imo introduces. This same story goes for RabbitMQ, I need to setup RabbitMQ in HA (which is fine) but I need to setup a Virtual IP for it as well just so Nova can correctly connect to it. Without the VIP the hole point of setting it up in a cluster is useless as you still will point nova to one single host. It shouldn't be that hard to make Nova listen to multiple mysql addresses and rabbitmq hosts in a cluster? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Florian Haas [mailto:flor...@hastexo.com] To: openstack@lists.launchpad.net Sent: Fri, 17 Feb 2012 13:32:51 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? On Thu, Feb 16, 2012 at 3:28 PM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Hi Tom, Thank you very much for sharing your knowledge, this will help not only me! As I am still working on the MySQL part, you mentioned multi-master which I have running indeed, but I suppose there still will need to be some kind of entering point within the cluster. How did you managed this? Virtual cluster IP addresses and highly available MySQL (either DRBD or MySQL replication based). Pacemaker is very well capable of managing this. Of course, since you also quoted http://openlife.cc/blogs/2011/july/ultimate-mysql-high-availability-solution parts of which I wholeheartedly disagree with (as with all of Henrik's Pacemaker bashing, but he's of course entitled to his own opinion), then you might run away screaming from anything that barely mentions Pacemaker in passing. But in reality[1], Pacemaker is an extremely capable HA stack that would be well suited for this. Cheers, Florian [1] To be polite, I should say in my humble opinion, but my opinion on this really isn't humble at all, so I prefer being honest over being polite. :) -- Need help with High Availability? http://www.hastexo.com/now ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
Hi Darren, No, I am not looking at a performance increasment at this moment. Just trying to make the different components Nova is build on redundant. Of course a master/master setup of mysql / rabbitmq with a floating IP will probably work but it bugs me that Nova is not really build with flexibility of it components in mind. I understand it is impossible to cover every scenario and you can make things redundant into the infinitive but why doesn't have Nova any of this build in? Why do I need to setup a Virtual IP/floating IP where Nova simple could accept multiple locations and failover if the first location fails. Or even round-robin to distribute the load. All I am doing now is yet add another component that can possibly fail as well. Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Darren Birkett [mailto:darren.birk...@gmail.com] To: i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl] Cc: Florian Haas [mailto:flor...@hastexo.com], openstack@lists.launchpad.net Sent: Tue, 21 Feb 2012 12:51:11 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? Surely it depends what you want out of it? If just pure high availability (ie ability to continue functioning when primary node goes down), then doesn't a master/master with floating IP fit the bill? It sounds like what you're after is better performance/throughput rather than HA - the ability to write to multiple masters at the same time. Even here if nova *was* able to hit multiple databases, a master/master setup isn't going to suit, as each write to one master still needs to be replicated to the other so throughput is going to be the same. You'd then need to look to something like a native mysql cluster to get actual increased throughput. Darren On 21 February 2012 11:34, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Ho Florian, I respect your opinion of course and I am not here to start a fight. Point is, how I understand it Nova really isn't very flexible on High Availability terms. I have now a master/master mysql setup with a Virtual IP working just to work around the shortcomings Nova imo introduces. This same story goes for RabbitMQ, I need to setup RabbitMQ in HA (which is fine) but I need to setup a Virtual IP for it as well just so Nova can correctly connect to it. Without the VIP the hole point of setting it up in a cluster is useless as you still will point nova to one single host. It shouldn't be that hard to make Nova listen to multiple mysql addresses and rabbitmq hosts in a cluster? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. From: Florian Haas [mailto:flor...@hastexo.com] To: openstack@lists.launchpad.net Sent: Fri, 17 Feb 2012 13:32:51 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? On Thu, Feb 16, 2012 at 3:28 PM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Hi Tom, Thank you very much for sharing your knowledge, this will help not only me! As I am still working on the MySQL part, you mentioned multi-master which I have running indeed, but I suppose there still will need to be some kind of entering point within the cluster. How did you managed this? Virtual cluster IP addresses and highly available MySQL (either DRBD or MySQL replication based). Pacemaker is very well capable of managing
Re: [Openstack] Howto Nova setup with HA?
Hm, I agree on that. Yes applications should be designed for the cloud no question there. How far is Nova ready for availability zones? Can I make different zones in the US and EU and let them work together? Is there any guidance for this available? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Michael Stein [mailto:michael.st...@q9.com] To: Diego Parrilla Santamaría [mailto:diego.parrilla.santama...@gmail.com], i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl] Cc: darren.birk...@gmail.com, openstack@lists.launchpad.net Sent: Tue, 21 Feb 2012 14:52:50 +0100 Subject: RE: [Openstack] Howto Nova setup with HA? I think that the idea behind Nova and “cloud” computing, in general, is that applications, middle-ware, databases and other services, will be delivered across multiple availability zones and provide HA via application-layer replication. The concept of failure is assumed and designed for by any forward thinking “cloud” designer. I think less time need to be spent on preserving nodes that maintain the “holy grail” of functionality and availability and more time needs to be spent on making apps cloud-friendly. In the case described below, I agree with Diego, networking is the real challenge. However, with a GSLB solution and application-level state/data replication you should be able to keep your services running even with an entire site failure. _ From: openstack-bounces+mstein=q9@lists.launchpad.net [mailto:openstack-bounces+mstein=q9@lists.launchpad.net] On Behalf Of Diego Parrilla Santamaría Sent: Tuesday, February 21, 2012 7:56 AM To: i3D.net - Tristan van Bokkem Cc: darren.birk...@gmail.com; openstack@lists.launchpad.net Subject: Re: [Openstack] Howto Nova setup with HA? This is the kind of flame war I love... ;-) if you think the HA challenges in Openstack Nova are at MySQL and Rabbit level, you are not putting the focus on the right components. The biggest HA (and scalability) challenges in Nova are in the networking side. There are a lot of companies tackling the networking challenges of Nova (BigSwitch, Nicira, Cisco, etc...), and I would like to see how they evolve in the future and how they will address this problem. Right now, only promises. Fight! -- Diego Parrilla CEO www.stackops.com | diego.parri...@stackops.com | +34 649 94 43 29 | skype:diegoparrilla On Tue, Feb 21, 2012 at 1:04 PM, i3D.net - Tristan van Bokkem tristanvanbok...@i3d.nl wrote: Hi Darren, No, I am not looking at a performance increasment at this moment. Just trying to make the different components Nova is build on redundant. Of course a master/master setup of mysql / rabbitmq with a floating IP will probably work but it bugs me that Nova is not really build with flexibility of it components in mind. I understand it is impossible to cover every scenario and you can make things redundant into the infinitive but why doesn't have Nova any of this build in? Why do I need to setup a Virtual IP/floating IP where Nova simple could accept multiple locations and failover if the first location fails. Or even round-robin to distribute the load. All I am doing now is yet add another component that can possibly fail as well. Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte
Re: [Openstack] Howto Nova setup with HA?
Yes, thank you. Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Darren Birkett [mailto:darren.birk...@gmail.com] To: Michael Stein [mailto:michael.st...@q9.com] Cc: Diego Parrilla Santamaría [mailto:diego.parrilla.santama...@gmail.com], i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl], openstack@lists.launchpad.net Sent: Tue, 21 Feb 2012 15:36:23 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? 2012/2/21 Michael Stein michael.st...@q9.com: I think that the idea behind Nova and “cloud” computing, in general, is that applications, middle-ware, databases and other services, will be delivered across multiple availability zones and provide HA via application-layer replication. The concept of failure is assumed and designed for by any forward thinking “cloud” designer. I think less time need to be spent on preserving nodes that maintain the “holy grail” of functionality and availability and more time needs to be spent on making apps cloud-friendly. Agreed, but you're talking about HA at a different layer - within the applications that will sit on top of the cloud infrastructure. Whilst that's absolutely valid, and a message that needs to be spread more amongst those designing apps for the cloud, it's not addressing the concerns of the OP. We still need to be thinking about HA of the cloud infrastructure components themselves, so that there are less failures for the app designers to have to tolerate in the first place. Darren ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
Any more thoughts about this subject? Ask? Vish? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Alexis Richardson [mailto:ale...@rabbitmq.com] To: Major Hayden [mailto:major.hay...@rackspace.com] Cc: i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl], openstack@lists.launchpad.net [mailto:openstack@lists.launchpad.net], Ask Solem [mailto:a...@rabbitmq.com] Sent: Tue, 14 Feb 2012 14:47:13 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? Major, The current set-up of HA in Openstack is borked in several ways. Ask Solem, cc'd, is going to write up a note helping people to do it properly. alexis On Tue, Feb 14, 2012 at 1:34 PM, Major Hayden major.hay...@rackspace.com wrote: On Feb 14, 2012, at 2:07 AM, i3D.net - Tristan van Bokkem wrote: So, we can run MySQL in master-master mode on multiple hosts, we can run nova-api on serveral hosts and load balance those and RabbitMQ has a cluster ha setup as well but is this the way to go? I can't find a clear answer to this. I am hoping one can shine some light on this! If you want to make things a little simpler, an active/passive MySQL setup with pacemaker, corosync, and DRBD will work quite well. ClusterLabs has a pretty decent howto on their site: http://www.clusterlabs.org/wiki/DRBD_MySQL_HowTo As for RabbitMQ clustering, I worked with it for several days and had quite a few problems with data consistency and availability. We eventually settled on the legacy H/A method for RabbitMQ using pacemaker and DRBD. There's a pretty good walkthrough on RabbitMQ's site: http://www.rabbitmq.com/pacemaker.html -- Major Hayden ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] Howto Nova setup with HA?
Interesting. Could you point us to the github location? Anyway, ive have done some further research myself and find this very interesting mysql replication setup done by wsrep (https://launchpad.net/wsrep) and galeria (https://launchpad.net/galera/) where mysql will run as a Synchronous Multi-Master replication setup. I have this running and works perfect but the problem now is. I need to add some kind of loadbalancing setup as nova can only point to one mysql location (i.e. --sql_connection=) So if I would point it to one of the mysql nodes in the cluster and that one fails everything fails. Adding a loadbalancer and point it to that will keep my mysql alive. The main issue with this is, we are adding another point where we have yet to configure and maintain several more nodes. I would like to raise the question if it would be possible for nova to accept multiple mysql connection points and therefor drop the need for extra loadbalancer component and reducing complexity. Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Jay Pipes [mailto:jaypi...@gmail.com] To: Alexis Richardson [mailto:ale...@rabbitmq.com] Cc: openstack@lists.launchpad.net Sent: Thu, 16 Feb 2012 13:11:36 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? Yup, absolutely... we've been a bit slammed working on getting the project up and running, but everything is checked in to GitHub... just needs a bit of clean up, that's all. Best, -jay On 02/16/2012 07:10 AM, Alexis Richardson wrote: It would be great to see the cookbooks if they are available, since others doing that setups might want such a thing. On Thu, Feb 16, 2012 at 11:51 AM, Jay Pipesjaypi...@gmail.com wrote: FWIW, we're using DRBD + pacemaker in active/passive setups for both the MySQL and RabbitMQ setup in the trystack.org (the artist formerly known as FreeCloud) project, and it's working quite well and is pretty simple to set up. Nati Ueno could give some more details on the setup and the Chef cookbooks used... Best, -jay On 02/16/2012 03:46 AM, i3D.net - Tristan van Bokkem wrote: Any more thoughts about this subject? Ask? Vish? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net mailto:tristanvanbok...@i3d.net E-mail Support: i...@i3d.netmailto:i...@i3d.net E-mail NOC: n...@i3d.netmailto:n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. *From:* Alexis Richardson [mailto:ale...@rabbitmq.com] *To:* Major Hayden [mailto:major.hay...@rackspace.com] *Cc:* i3D.net - Tristan van Bokkem [mailto:tristanvanbok...@i3d.nl], openstack@lists.launchpad.net [mailto:openstack@lists.launchpad.net], Ask Solem [mailto:a...@rabbitmq.com] *Sent:* Tue, 14 Feb 2012 14:47:13 +0100 *Subject:* Re: [Openstack] Howto Nova setup with HA? Major, The current set-up of HA in Openstack is borked in several ways. Ask Solem, cc'd, is going to write up a note helping people to do it properly. alexis On Tue, Feb 14, 2012 at 1:34 PM, Major Hayden major.hay...@rackspace.commailto:major.hay...@rackspace.com wrote: On Feb 14, 2012, at 2:07 AM, i3D.net - Tristan van Bokkem wrote: So, we can run MySQL in master-master mode on multiple hosts, we can run nova-api on serveral hosts and load balance those and RabbitMQ has a cluster ha setup as well but is this the way
Re: [Openstack] Howto Nova setup with HA?
Hi Tom, Thank you very much for sharing your knowledge, this will help not only me! As I am still working on the MySQL part, you mentioned multi-master which I have running indeed, but I suppose there still will need to be some kind of entering point within the cluster. How did you managed this? Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies. _ From: Tom Ellis [mailto:tom.el...@canonical.com] To: openstack@lists.launchpad.net Sent: Thu, 16 Feb 2012 15:09:15 +0100 Subject: Re: [Openstack] Howto Nova setup with HA? On 16/02/12 08:46, i3D.net - Tristan van Bokkem wrote: Any more thoughts about this subject? Ask? Vish? The team I work in has looked at a number of methods for high availability within the Diablo release and I've included some notes below, hope this helps. Font-end API servers * load balanced with h/w load balancer * use s/w LB for smaller deployments * run nova-scheduler on each MySQL DB * multi-master configuration * alternative: drbd + pacemaker in active/passive RabbitMQ service * Pacemaker with Active-passive configuration * Based on example from RabbitMQ site (already mentioned by someone else) * Virtual IP for the service - used for rabbitmq config in nova.conf * I think there are some corner cases where messages could be dropped during failover and I believe later RabbitMQ versions support full multi-master but require some client side changes - is there any plans to support this? nova-volume service * Current weakness in the HA setup, unless you are willing to use iscsi tgtd with DRBD. I believe this would still have some problems when failing over with the initiators that are logged in. * I'm hoping this will pan out for essex with some of the storage vendors committing nova-volume support via their API's. Glance * Run on multiple servers * Use another VIP in your pacemaker setup or load-balancer * Use swift as backend storage Compute servers * Each run their own copy of nova-api (only instances running on the node use this) * nova-network (multi-host configuration) with private network Swift * Run swift-proxy across all swift-storage nodes on a small setup Regards, Tom ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
[Openstack] Howto Nova setup with HA?
Hi Stackers, It seems running Openstack components in High Availability hasn't been really a focus point lately, am I right? The general docs don't really mention HA except for nova-network. So I did some resource on how to run Nova in a High Availability and have some questions about it: The docs guides you on how to setup one cloud controller (running MySQL, nova-api, RabbitMQ etc.) and 2+n nodes for nova compute/network. But it does not mention how to make the cloud controller redundant. If the cloud controller brakes we have a serious problem! So, we can run MySQL in master-master mode on multiple hosts, we can run nova-api on serveral hosts and load balance those and RabbitMQ has a cluster ha setup as well but is this the way to go? I can't find a clear answer to this. I am hoping one can shine some light on this! Best regards, Tristan van Bokkem Datacenter Operations Contact: E-mail Personal: tristanvanbok...@i3d.net E-mail Support: i...@i3d.net E-mail NOC: n...@i3d.net Website: http://www.i3d.net Office: Interactive 3D B.V. Meent 93b 3011 JG Rotterdam The Netherlands Visit www.smartdc.net – SmartDC is our in-house 36,000 sq. ft. datacenter in Rotterdam, The Netherlands. High density hosting – multiple fiber carriers in-house – Level3 PoP. Interactive 3D (i3D.net) is a company registered in The Netherlands at Meent 93b, Rotterdam. Registration #: 14074337 - VAT # NL 8202.63.886.B01. Interactive 3D (i3D.net) is CDSA certified on content protection and security. We are ranked in the Deloitte Technology Fast 50 as one of the fastest growing technology companies.___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
