Sorry for the long delay in getting back to you on this, Dan. Comments inline...
On Tue, Dec 13, 2011 at 10:26 AM, Dan Prince <dan.pri...@rackspace.com> wrote: > Hi Jay, > > The tests should not be running concurrently. OK, then the issue must be the fsync() timing on the servers... > We currently have 4 Natty Cloud Servers configured as unit test workers. The > machines are shared between nova/glance/keystone. ok. > The unit test runner is pretty simple: > > https://github.com/dprince/smokestack/blob/master/app/templates/unittest_runner.sh.erb Yep, nothing unusual there... though I would check with Jim Blair about the use of run_tests.sh -V -f in there. I believe Jim made some improvements to the virtualenv setup that eliminates the need to refresh the virtualenv (the -f flag) every time it is run... > Would using a unique FAKE_FILESYSTEM_ROOTDIR for each test run or test help > out here? Perhaps prefixing each one w/ /tmp/glance-tests so that we could > also add an explicit tmp dir cleanup command in the test suite runners as > well. Yep, using a unique tmp dir for that would work (that's what the functional tests in Glance do). I've added a bug report to do this: https://bugs.launchpad.net/glance/+bug/913608 > In the meantime I'm happy to add an explicit cleanup for /tmp/glance-tests > to make sure we have a clean slate for each test run. I think the solution to the above bug should handle these test failures. I should have a patch proposed shortly. Cheers! -jay > > > Dan > > > > -----Original Message----- > From: "Jay Pipes" <jaypi...@gmail.com> > Sent: Tuesday, December 13, 2011 7:14am > To: "Dan Prince" <dan.pri...@rackspace.com> > Cc: "Openstack" <openstack@lists.launchpad.net> > Subject: Re: [Openstack] unit and integration tests results for Gerrit > > > Thanks Dan, looks like a great start on this. Not sure what's going on > with the unit test runs in Glance, though... As an example, see: > > http://smokestack.openstack.org/?go=/jobs/5600 > > There are dozens of errors like this: > > ====================================================================== > ERROR: test_add_member (glance.tests.unit.test_api.TestGlanceAPI) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/tmp/tmp.ZaWdEoPfhP/glance_source/glance/tests/unit/test_api.py", > line 1937, in setUp > stubs.stub_out_filesystem_backend() > File "/tmp/tmp.ZaWdEoPfhP/glance_source/glance/tests/stubs.py", line > 63, in stub_out_filesystem_backend > os.mkdir(FAKE_FILESYSTEM_ROOTDIR) > OSError: [Errno 17] File exists: '/tmp/glance-tests' > > That FAKE_FILESYSTEM_ROOTDIR is cleaned up in the tearDown() method of > API unit tests, and these tests run perfectly fine for me locally. I > was thinking that one of the following things may be occurring: > > * The unlink() of the FAKE_FILESYSTEM_ROOTDIR is not fsync'ing fast > enough, resulting in setUp()'s call to create the /tmp/glance-tests > directory is stumbling over itself > * The tests are being run in parallel somehow? > > I've seen the fsync behaviour cause havoc in some of the image cache > tests before, and the solution ended up putting a small wait loop in > the test code to wait until disk buffers were flushed and a cache file > was fully removed from the filesystem. That may be happening here? > These are being run on Cloud Servers, right? > > /me just trying to figure out why tests would run differently in > smokestack than Jenkins or locally... > > Thanks in advance for any insight. > > Cheers, > -jay _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp