Hi again Joe, (+ list)
On 11/04/15 02:00, joehuang wrote:
Hi, Neil,
See inline comments.
Best Regards
Chaoyi Huang
________________________________________
From: Neil Jerram [neil.jer...@metaswitch.com]
Sent: 09 April 2015 23:01
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] Neutron scaling datapoints?
Hi Joe,
Many thanks for your reply!
On 09/04/15 03:34, joehuang wrote:
Hi, Neil,
From theoretic, Neutron is like a "broadcast" domain, for example, enforcement of DVR and
security group has to touch each regarding host where there is VM of this project resides. Even using SDN
controller, the "touch" to regarding host is inevitable. If there are plenty of physical hosts, for
example, 10k, inside one Neutron, it's very hard to overcome the "broadcast storm" issue under
concurrent operation, that's the bottleneck for scalability of Neutron.
I think I understand that in general terms - but can you be more
specific about the broadcast storm? Is there one particular message
exchange that involves broadcasting? Is it only from the server to
agents, or are there 'broadcasts' in other directions as well?
[[joehuang]] for example, L2 population, Security group rule update, DVR route
update. Both direction in different scenario.
Thanks. In case it's helpful to see all the cases together,
sync_routers (from the L3 agent) was also mentioned in other part of
this thread. Plus of course the liveness reporting from all agents.
(I presume you are talking about control plane messages here, i.e.
between Neutron components. Is that right? Obviously there can also be
broadcast storm problems in the data plane - but I don't think that's
what you are talking about here.)
[[joehuang]] Yes, controll plane here.
Thanks for confirming that.
We need layered architecture in Neutron to solve the "broadcast domain" bottleneck of
scalability. The test report from OpenStack cascading shows that through layered architecture
"Neutron cascading", Neutron can supports up to million level ports and 100k level
physical hosts. You can find the report here:
http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers
Many thanks, I will take a look at this.
It was very interesting, thanks. And by following through your links I
also learned more about Nova cells, and about how some people question
whether we need any kind of partitioning at all, and should instead
solve scaling/performance problems in other ways... It will be
interesting to see how this plays out.
I'd still like to see more information, though, about how far people
have scaled OpenStack - and in particular Neutron - as it exists today.
Surely having a consensus set of current limits is an important input
into any discussion of future scaling work.
For example, Kevin mentioned benchmarking where the Neutron server
processed a liveness update in <50ms and a sync_routers in 300ms.
Suppose, the liveness update time was 50ms (since I don't know in detail
what that < means) and agents report liveness every 30s. Does that mean
that a single Neutron server can only support 600 agents?
I'm also especially interested in the DHCP agent, because in Calico we
have one of those on every compute host. We've just run tests which
appeared to be hitting trouble from just 50 compute hosts onwards, and
apparently because of DHCP agent communications. We need to continue
looking into that and report findings properly, but if anyone already
has any insights, they would be much appreciated.
Many thanks,
Neil
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
