[Yahoo-eng-team] [Bug 1384660] Re: Idle rpc traffic with a large number of instances causes failures
** Changed in: neutron Status: Incomplete => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1384660 Title: Idle rpc traffic with a large number of instances causes failures Status in Ubuntu Cloud Archive: Fix Released Status in neutron: Fix Released Bug description: OpenStack Juno (Neutron ML2+OVS/l2pop/neutron security groups), Ubuntu 14.04 500 compute node cloud, running 4.5k active instances (can't get it any further right now). As the number of instances in the cloud increases, the idle loading on the neutron-server servers (4 of them all with 4 cores/8 threads and a suitable *_worker configuration) increases from nothing to 30; The db call get_port_and_sgs is being serviced around 10 times per second on each server at this point. Other things are also happening - I've attached the last 1000 lines of the server log with debug enabled. The result is that its no longer possible to create new instances, as the rpc calls and api thread just don't get onto CPU, resulting in VIF plugging timeouts on compute nodes, and ERROR'ed instances. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: neutron-common 1:2014.2-0ubuntu1~cloud0 [origin: Canonical] ProcVersionSignature: User Name 3.13.0-35.62-generic 3.13.11.6 Uname: Linux 3.13.0-35-generic x86_64 ApportVersion: 2.14.1-0ubuntu3.5 Architecture: amd64 CrashDB: { "impl": "launchpad", "project": "cloud-archive", "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml;, } Date: Thu Oct 23 10:22:14 2014 PackageArchitecture: all SourcePackage: neutron UpgradeStatus: No upgrade log present (probably fresh install) modified.conffile..etc.neutron.api.paste.ini: [deleted] modified.conffile..etc.neutron.fwaas.driver.ini: [deleted] modified.conffile..etc.neutron.l3.agent.ini: [deleted] modified.conffile..etc.neutron.neutron.conf: [deleted] modified.conffile..etc.neutron.policy.json: [deleted] modified.conffile..etc.neutron.rootwrap.conf: [deleted] modified.conffile..etc.neutron.rootwrap.d.debug.filters: [deleted] modified.conffile..etc.neutron.rootwrap.d.ipset.firewall.filters: [deleted] modified.conffile..etc.neutron.rootwrap.d.iptables.firewall.filters: [deleted] modified.conffile..etc.neutron.rootwrap.d.l3.filters: [deleted] modified.conffile..etc.neutron.rootwrap.d.vpnaas.filters: [deleted] modified.conffile..etc.neutron.vpn.agent.ini: [deleted] modified.conffile..etc.sudoers.d.neutron.sudoers: [deleted] To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1384660/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1384660] Re: Idle rpc traffic with a large number of instances causes failures
I believe the optimizations detail in this change have now landed in = Juno; marking fix released. ** Changed in: cloud-archive Status: New = Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1384660 Title: Idle rpc traffic with a large number of instances causes failures Status in Ubuntu Cloud Archive: Fix Released Status in OpenStack Neutron (virtual network service): Confirmed Bug description: OpenStack Juno (Neutron ML2+OVS/l2pop/neutron security groups), Ubuntu 14.04 500 compute node cloud, running 4.5k active instances (can't get it any further right now). As the number of instances in the cloud increases, the idle loading on the neutron-server servers (4 of them all with 4 cores/8 threads and a suitable *_worker configuration) increases from nothing to 30; The db call get_port_and_sgs is being serviced around 10 times per second on each server at this point. Other things are also happening - I've attached the last 1000 lines of the server log with debug enabled. The result is that its no longer possible to create new instances, as the rpc calls and api thread just don't get onto CPU, resulting in VIF plugging timeouts on compute nodes, and ERROR'ed instances. ProblemType: Bug DistroRelease: Ubuntu 14.04 Package: neutron-common 1:2014.2-0ubuntu1~cloud0 [origin: Canonical] ProcVersionSignature: User Name 3.13.0-35.62-generic 3.13.11.6 Uname: Linux 3.13.0-35-generic x86_64 ApportVersion: 2.14.1-0ubuntu3.5 Architecture: amd64 CrashDB: { impl: launchpad, project: cloud-archive, bug_pattern_url: http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml;, } Date: Thu Oct 23 10:22:14 2014 PackageArchitecture: all SourcePackage: neutron UpgradeStatus: No upgrade log present (probably fresh install) modified.conffile..etc.neutron.api.paste.ini: [deleted] modified.conffile..etc.neutron.fwaas.driver.ini: [deleted] modified.conffile..etc.neutron.l3.agent.ini: [deleted] modified.conffile..etc.neutron.neutron.conf: [deleted] modified.conffile..etc.neutron.policy.json: [deleted] modified.conffile..etc.neutron.rootwrap.conf: [deleted] modified.conffile..etc.neutron.rootwrap.d.debug.filters: [deleted] modified.conffile..etc.neutron.rootwrap.d.ipset.firewall.filters: [deleted] modified.conffile..etc.neutron.rootwrap.d.iptables.firewall.filters: [deleted] modified.conffile..etc.neutron.rootwrap.d.l3.filters: [deleted] modified.conffile..etc.neutron.rootwrap.d.vpnaas.filters: [deleted] modified.conffile..etc.neutron.vpn.agent.ini: [deleted] modified.conffile..etc.sudoers.d.neutron.sudoers: [deleted] To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1384660/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp