On Jun 29, 2012, at 3:27 PM, Jay Pipes wrote: > Hi all, > > OK, so while we've done a commendable job so far in adding a large *quantity* > of tests to Tempest, one ramification of this increase in tests has been the > large increase in runtime for the whole Tempest test suite. Nowadays, the > whole suite is taking upwards of 45 minutes to complete -- and that's with > nearly 70 skipped tests! > > We need to be more careful about *how* we write tests in Tempest, and we need > to be mindful of how long each test added takes to run and the resources each > test creates/modifies/destroys. > > In many cases, we've allowed tests to be added to Tempest that take a > significant amount of time to run -- and these tests often can and should be > combined into fewer test methods that use fewer resources. >
[snip] > We need to consider these types of things when we write tests for Tempest. > The point should not be to just increase the *quantity* of tests, but rather > to increase the testing while keeping the runtime of tempest down. > As it happens, the problem of reducing regression testing time (by reducing the number of tests in an intelligent way) is one of the most heavily studied areas in software engineering research. I don't have any personal expertise in this area, but if you do a search for "regression test selection" on Google Scholar, you'll find a ton of stuff on this. Gregg Rothermel at University of Nebraska-Lincoln is one big name in this area. Take care, Lorin -- Lorin Hochstein Lead Architect - Cloud Services Nimbis Services, Inc. www.nimbisservices.com
-- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp
