-----Original Message----- From: Attila Fazekas [mailto:afaze...@redhat.com] Sent: Monday, April 13, 2015 3:19 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
----- Original Message ----- > From: "joehuang" <joehu...@huawei.com> > To: "OpenStack Development Mailing List (not for usage questions)" > <openstack-dev@lists.openstack.org> > Sent: Sunday, April 12, 2015 1:20:48 PM > Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints? > > > > Hi, Kevin, > > > > I assumed that all agents are connected to same IP address of > RabbitMQ, then the connection will exceed the port ranges limitation. > https://news.ycombinator.com/item?id=1571300 "TCP connections are identified by the (src ip, src port, dest ip, dest port) tuple." "The server doesn't need multiple IPs to handle > 65535 connections. All the server connections to a given IP are to the same port. For a given client, the unique key for an http connection is (client-ip, PORT, server-ip, 80). The only number that can vary is PORT, and that's a value on the client. So, the client is limited to 65535 connections to the server. But, a second client could also have another 65K connections to the same server-ip:port." [[joehuang]] Sorry, long time not writing socket based app, I may make a mistake for HTTP server to spawn a thread to handle a new connection. I'll check again. > > For a RabbitMQ cluster, for sure the client can connect to any one of > member in the cluster, but in this case, the client has to be designed > in fail-safe > manner: the client should be aware of the cluster member failure, and > reconnect to other survive member. No such mechnism has been > implemented yet. > > > > Other way is to use LVS or DNS based like load balancer, or something else. > If you put one load balancer ahead of a cluster, then we have to take > care of the port number limitation, there are so many agents will > require connection concurrently, 100k level, and the requests can not be > rejected. > > > > Best Regards > > > > Chaoyi Huang ( joehuang ) > > > > From: Kevin Benton [blak...@gmail.com] > Sent: 12 April 2015 9:59 > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints? > > The TCP/IP stack keeps track of connections as a combination of IP + > TCP port. The two byte port limit doesn't matter unless all of the > agents are connecting from the same IP address, which shouldn't be the > case unless compute nodes connect to the rabbitmq server via one IP > address running port address translation. > > Either way, the agents don't connect directly to the Neutron server, > they connect to the rabbit MQ cluster. Since as many Neutron server > processes can be launched as necessary, the bottlenecks will likely > show up at the messaging or DB layer. > > On Sat, Apr 11, 2015 at 6:46 PM, joehuang < joehu...@huawei.com > wrote: > > > > > > As Kevin talking about agents, I want to remind that in TCP/IP stack, > port ( not Neutron Port ) is a two bytes field, i.e. port ranges from > 0 ~ 65535, supports maximum 64k port number. > > > > " above 100k managed node " means more than 100k L2 agents/L3 > agents... will be alive under Neutron. > > > > Want to know the detail design how to support 99.9% possibility for > scaling Neutron in this way, and PoC and test would be a good support for > this idea. > > > > "I'm 99.9% sure, for scaling above 100k managed node, we do not really > need to split the openstack to multiple smaller openstack, or use > significant number of extra controller machine." > > > > Best Regards > > > > Chaoyi Huang ( joehuang ) > > > > From: Kevin Benton [ blak...@gmail.com ] > Sent: 11 April 2015 12:34 > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints? > > Which periodic updates did you have in mind to eliminate? One of the > few remaining ones I can think of is sync_routers but it would be > great if you can enumerate the ones you observed because eliminating > overhead in agents is something I've been working on as well. > > One of the most common is the heartbeat from each agent. However, I > don't think we can't eliminate them because they are used to determine > if the agents are still alive for scheduling purposes. Did you have > something else in mind to determine if an agent is alive? > > On Fri, Apr 10, 2015 at 2:18 AM, Attila Fazekas < afaze...@redhat.com > > > wrote: > > > I'm 99.9% sure, for scaling above 100k managed node, we do not really > need to split the openstack to multiple smaller openstack, or use > significant number of extra controller machine. > > The problem is openstack using the right tools SQL/AMQP/(zk), but in a > wrong way. > > For example.: > Periodic updates can be avoided almost in all cases > > The new data can be pushed to the agent just when it needed. > The agent can know when the AMQP connection become unreliable (queue > or connection loose), and needs to do full sync. > https://bugs.launchpad.net/neutron/+bug/1438159 > > Also the agents when gets some notification, they start asking for > details via the AMQP -> SQL. Why they do not know it already or get it > with the notification ? > > > ----- Original Message ----- > > From: "Neil Jerram" < neil.jer...@metaswitch.com > > > To: "OpenStack Development Mailing List (not for usage questions)" < > > openstack-dev@lists.openstack.org > > > Sent: Thursday, April 9, 2015 5:01:45 PM > > Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints? > > > > Hi Joe, > > > > Many thanks for your reply! > > > > On 09/04/15 03:34, joehuang wrote: > > > Hi, Neil, > > > > > > From theoretic, Neutron is like a "broadcast" domain, for example, > > > enforcement of DVR and security group has to touch each regarding > > > host where there is VM of this project resides. Even using SDN > > > controller, the "touch" to regarding host is inevitable. If there > > > are plenty of physical hosts, for example, 10k, inside one > > > Neutron, it's very hard to overcome the "broadcast storm" issue > > > under concurrent operation, that's the bottleneck for scalability of > > > Neutron. > > > > I think I understand that in general terms - but can you be more > > specific about the broadcast storm? Is there one particular message > > exchange that involves broadcasting? Is it only from the server to > > agents, or are there 'broadcasts' in other directions as well? > > > > (I presume you are talking about control plane messages here, i.e. > > between Neutron components. Is that right? Obviously there can also > > be broadcast storm problems in the data plane - but I don't think > > that's what you are talking about here.) > > > > > We need layered architecture in Neutron to solve the "broadcast domain" > > > bottleneck of scalability. The test report from OpenStack > > > cascading shows that through layered architecture "Neutron > > > cascading", Neutron can supports up to million level ports and > > > 100k level physical hosts. You can find the report here: > > > http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cas > > > cading-solution-to-support-1-million-v-ms-in-100-data-centers > > > > Many thanks, I will take a look at this. > > > > > "Neutron cascading" also brings extra benefit: One cascading > > > Neutron can have many cascaded Neutrons, and different cascaded > > > Neutron can leverage different SDN controller, maybe one is ODL, > > > the other one is OpenContrail. > > > > > > ----------------Cascading Neutron------------------- / \ > > > --cascaded Neutron-- --cascaded Neutron----- > > > | | > > > ---------ODL------ ----OpenContrail-------- > > > > > > > > > And furthermore, if using Neutron cascading in multiple data > > > centers, the DCI controller (Data center inter-connection > > > controller) can also be used under cascading Neutron, to provide > > > NaaS ( network as a service ) across data centers. > > > > > > ---------------------------Cascading > > > Neutron-------------------------- / | \ --cascaded Neutron-- -DCI > > > controller- --cascaded Neutron----- > > > | | | > > > ---------ODL------ | ----OpenContrail-------- > > > | > > > --(Data center 1)-- --(DCI networking)-- --(Data center 2)-- > > > > > > Is it possible for us to discuss this in OpenStack Vancouver summit? > > > > Most certainly, yes. I will be there from mid Monday afternoon > > through to end Friday. But it will be my first summit, so I have no > > idea yet as to how I might run into you - please can you suggest! > > > > > Best Regards > > > Chaoyi Huang ( Joe Huang ) > > > > Regards, > > Neil > > > > ____________________________________________________________________ > > ______ OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > ______________________________________________________________________ > ____ OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > -- > Kevin Benton > > ______________________________________________________________________ > ____ OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > -- > Kevin Benton > > ______________________________________________________________________ > ____ OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev