Re: [Openstack] How to deploy OpenStack on thousands of nodes?

2013-06-26 Thread Tim Bell
I'd suggest to have a look at the openstack Operations Guide which contains information on scaling approaches (http://docs.openstack.org/ops/) In particular, the use of cells is a good way to build large scale infrastructures (see

Re: [Openstack] How to deploy OpenStack on thousands of nodes?

2013-06-26 Thread Brent Roskos
Kylin, I think there is some confusion as to the term broadcast. Many of the Rabbit docs describe the delivery of a message from one publisher to multiple subscribers as a 'broadcast'. This is not to be confused with a network broadcast where traffic is sent over the network broadcast address.

Re: [Openstack] How to deploy OpenStack on thousands of nodes?

2013-06-26 Thread Sg Kylin
Hi Brent, Thanks very much for your sharing of your experience. I want to clarify that we are indeed trying to deploy OpenStack on thousands of nodes. The first block we met was keepalived based HA which needs multicast. Thus we are reviewing our network topology design again. At the same time we

Re: [Openstack] How to deploy OpenStack on thousands of nodes?

2013-06-26 Thread Brent Roskos
Kylin, I've previously used ECMP to balance traffic to Openstack components that scale laterally like nova-api. This was fairly easy to set up and worked well for what we needed where state was not needed. I've also used keepalived for services that did not scale laterally. In this case I put

[Openstack] How to deploy OpenStack on thousands of nodes?

2013-06-25 Thread Sg Kylin
Hi All, We are currently trying to deploy OpenStack on thousands of nodes. We are using Grizzly stable version and Ubuntu 12.04.2. However, the big problem we meet now is the network topology. If we want to use HA (haproxy + keepalived) for the controller nodes on which *-apis are running as well

Re: [Openstack] How to deploy OpenStack on thousands of nodes?

2013-06-25 Thread Brent Roskos
I'll speak to your rabbit question: By default rabbit uses tcp port 5672 for communication.. tcp can certainly cross subnet boundaries and be routed without issue. I suggest you do some network troubleshooting; ping your rabbit server then telnet to port 5672 on the rabbit server from hosts on

Re: [Openstack] How to deploy OpenStack on thousands of nodes?

2013-06-25 Thread Brent Roskos
By default rabbit uses tcp port 5672 for communication.. tcp can certainly cross subnet boundaries and be routed without issue. I suggest you do some network troubleshooting; ping your rabbit server then telnet to port 5672 on the rabbit server from hosts on the other subnets. Check your

Re: [Openstack] How to deploy OpenStack on thousands of nodes?

2013-06-25 Thread Sg Kylin
Hi Brent, Thanks for your reply! But we are afraid that Rabbitmq needs broadcast to work correctly and usually broadcast is not available in cross-subnets deployments. That is what we are worrying about... Best, Kylin CG 2013/6/26 Brent Roskos brent.ros...@solinea.com By default rabbit