Another way to tackle it would be to create a dedicated tenant for those tests, then the quota won't interact with anything else.
On 3 January 2014 10:35, Miguel Angel Ajo Pelayo <mangel...@redhat.com> wrote: > Hi Salvatore!, > > Good work on this. > > About the quota limit tests, I believe they may be unit-tested, > instead of functionally tested. > > When running those tests in parallel with any other tests that rely > on having ports, networks or subnets available into quota, they have > high chances of making those other tests fail. > > Cheers, > Miguel Ángel Ajo > > > > ----- Original Message ----- >> From: "Kyle Mestery" <mest...@siliconloons.com> >> To: "OpenStack Development Mailing List (not for usage questions)" >> <openstack-dev@lists.openstack.org> >> Sent: Thursday, January 2, 2014 7:53:05 PM >> Subject: Re: [openstack-dev] [Neutron][qa] Parallel testing update >> >> Thanks for the updates here Salvatore, and for continuing to push on >> this! This is all great work! >> >> On Jan 2, 2014, at 6:57 AM, Salvatore Orlando <sorla...@nicira.com> wrote: >> > >> > Hi again, >> > >> > I've now run the experimental job a good deal of times, and I've filed bugs >> > for all the issues which came out. >> > Most of them occurred no more than once among all test execution (I think >> > about 30). >> > >> > They're all tagged with neutron-parallel [1]. for ease of tracking, I've >> > associated all the bug reports with neutron, but some are probably more >> > tempest or nova issues. >> > >> > Salvatore >> > >> > [1] https://bugs.launchpad.net/neutron/+bugs?field.tag=neutron-parallel >> > >> > >> > On 27 December 2013 11:09, Salvatore Orlando <sorla...@nicira.com> wrote: >> > Hi, >> > >> > We now have several patches under review which improve a lot how neutron >> > handles parallel testing. >> > In a nutshell, these patches try to ensure the ovs agent processes new, >> > removed, and updated interfaces as soon as possible, >> > >> > These patches are: >> > https://review.openstack.org/#/c/61105/ >> > https://review.openstack.org/#/c/61964/ >> > https://review.openstack.org/#/c/63100/ >> > https://review.openstack.org/#/c/63558/ >> > >> > There is still room for improvement. For instance the calls from the agent >> > into the plugins might be consistently reduced. >> > However, even if the above patches shrink a lot the time required for >> > processing a device, we are still hitting a hard limit with the execution >> > ovs commands for setting local vlan tags and clearing flows (or adding the >> > flow rule for dropping all the traffic). >> > In some instances this commands slow down a lot, requiring almost 10 >> > seconds to complete. This adds a delay in interface processing which in >> > some cases leads to the hideous SSH timeout error (the same we see with >> > bug 1253896 in normal testing). >> > It is also worth noting that when this happens sysstat reveal CPU usage is >> > very close to 100% >> > >> > From the neutron side there is little we can do. Introducing parallel >> > processing for interface, as we do for the l3 agent, is not actually a >> > solution, since ovs-vswitchd v1.4.x, the one executed on gate tests, is >> > not multithreaded. If you think the situation might be improved by >> > changing the logic for handling local vlan tags and putting ports on the >> > dead vlan, I would be happy to talk about that. >> > On my local machines I've seen a dramatic improvement in processing times >> > by installing ovs 2.0.0, which has a multi-threaded vswitchd. Is this >> > something we might consider for gate tests? Also, in order to reduce CPU >> > usage on the gate (and making tests a bit faster), there is a tempest >> > patch which stops creating and wiring neutron routers when they're not >> > needed: https://review.openstack.org/#/c/62962/ >> > >> > Even in my local setup which succeeds about 85% of times, I'm still seeing >> > some occurrences of the issue described in [1], which at the end of the >> > day seems a dnsmasq issue. >> > >> > Beyond the 'big' structural problem discussed above, there are some minor >> > problems with a few tests: >> > >> > 1) test_network_quotas.test_create_ports_until_quota_hit fails about 90% >> > of times. I think this is because the test itself should be made aware of >> > parallel execution and asynchronous events, and there is a patch for this >> > already: https://review.openstack.org/#/c/64217 >> > >> > 2) test_attach_interfaces.test_create_list_show_delete_interfaces fails >> > about 66% of times. The failure is always on an assertion made after >> > deletion of interfaces, which probably means the interface is not deleted >> > within 5 seconds. I think this might be a consequence of the higher load >> > on the neutron service and we might try to enable multiple workers on the >> > gate to this aim, or just increase the tempest timeout. On a slightly >> > different note, allow me to say that the way assertion are made on this >> > test might be improved a bit. So far one has to go through the code to see >> > why the test failed. >> > >> > Thanks for reading this rather long message. >> > Regards, >> > Salvatore >> > >> > [1] https://lists.launchpad.net/openstack/msg23817.html >> > >> > >> > >> > >> > On 2 December 2013 22:01, Kyle Mestery (kmestery) <kmest...@cisco.com> >> > wrote: >> > Yes, this is all great Salvatore and Armando! Thank you for all of this >> > work >> > and the explanation behind it all. >> > >> > Kyle >> > >> > On Dec 2, 2013, at 2:24 PM, Eugene Nikanorov <enikano...@mirantis.com> >> > wrote: >> > >> > > Salvatore and Armando, thanks for your great work and detailed >> > > explanation! >> > > >> > > Eugene. >> > > >> > > >> > > On Mon, Dec 2, 2013 at 11:48 PM, Joe Gordon <joe.gord...@gmail.com> >> > > wrote: >> > > >> > > On Dec 2, 2013 9:04 PM, "Salvatore Orlando" <sorla...@nicira.com> wrote: >> > > > >> > > > Hi, >> > > > >> > > > As you might have noticed, there has been some progress on parallel >> > > > tests for neutron. >> > > > In a nutshell: >> > > > * Armando fixed the issue with IP address exhaustion on the public >> > > > network [1] >> > > > * Salvatore has now a patch which has a 50% success rate (the last >> > > > failures are because of me playing with it) [2] >> > > > * Salvatore is looking at putting back on track full isolation [3] >> > > > * All the bugs affecting parallel tests can be queried here [10] >> > > > * This blueprint tracks progress made towards enabling parallel testing >> > > > [11] >> > > > >> >> >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Robert Collins <rbtcoll...@hp.com> Distinguished Technologist HP Converged Cloud _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev