On 07/12/11 15:09, Shawn Walker wrote:
On 07/08/11 01:33, Brock Pytlik wrote:
Webrev:
https://cr.opensolaris.org/action/browse/pkg/bpytlik/parallel-test-v1
Bugs:
18636 a single test suite run should be parallelizable
18571 test suite uses obsolete unittest function makeSuite
18616 fakeroot_repo should die
src/setup.py:
line 708: maybe 'jobs' instead?
lines 772, 811: docstring so we know the purpose?
lines 1230-1256: some comments would be helpful
line 1416: s/not/no/
line 1429: s/esimate/estimate/
lines 1479-1485: space after ':' in strings please
These all seem reasonable.
-------
So after applying the patch to gate tip, there are some regressions /
things I noticed:
* output from a simple 'sudo python run.py -v' is no longer updated
as each test runs, instead it appears to be updated only after
each unit test (e.g. api.t_version) runs; this leaves the user
sitting a long time without seeing any progress output
Yeah, that's an unfortunate consequence of how I decided to parallelize
things and helps to keep lock contention at a minimum. If someone
believes that this needs to be fixed, I can fix it by making the
one-process case use the original code.
* when I interrupted the test run above, I got this traceback:
So in general, I spent a lot of time trying to handle the ctrl-c case as
best I could. What you see is as good as I can make it (other than that,
this wad was probably about done a week or more ago). If people feel
that the tracebacks on ctrl-c mean we don't want the feature, I'll
understand, or if someone wants to offer a patch to improve things, I'd
be thrilled to take it. I have had one more idea while writing this
reply. Given my recent experiences w/ signal handling in python, I don't
really think it'll work, but I'll try and see if it fixes anything.
^C
# Ran 158 tests in 178.591s - skipped 0 tests.
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/queues.py", line 242, in _feed
send(obj)
IOError: [Errno 4] Interrupted system call
All spawned processes should be terminated, now cleaning up directories.
Directories successfully removed.
* i also noticed that after the above, I was left with a zero-length
test log file named /tmp/tmpHQRcRv.pkg-test.log
That's a little strange, but I've never understood what that log's for
in the first place. I'll look into it.
* if I run just a single unit test successfully, I still see a
zero-length log flie on exist such as /tmp/tmpVclAYf.pkg-test.log;
I realise the earlier log message said it was logging there, but
it never seems to contain anything?
So the obvious question I'll ask (which I'll go off and test the next
time I work on it) is "What does it contain when the gate runs?"
* I then did another test run with -j 16 and interrupted it after a
while and saw a mammoth traceback that basically repeated this:
See above comments on ctrl-C.
...
Process Process-27:
File "/usr/lib/python2.6/multiprocessing/queues.py", line 89, in get
Traceback (most recent call last):
self._rlock.acquire()
KeyboardInterrupt
File "/usr/lib/python2.6/multiprocessing/queues.py", line 89, in get
test_suite = inq.get()
File "/usr/lib/python2.6/multiprocessing/queues.py", line 89, in get
self._rlock.acquire()
KeyboardInterrupt
self._rlock.acquire()
KeyboardInterrupt
Process Process-32:
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
self.run()
File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
self._target(*self._args, **self._kwargs)
File "./pkg5unittest.py", line 1235, in q_run
test_suite = inq.get()
File "/usr/lib/python2.6/multiprocessing/queues.py", line 89, in get
self._rlock.acquire()
KeyboardInterrupt
Process Process-23:
test_suite = inq.get()
self._target(*self._args, **self._kwargs)
File "./pkg5unittest.py", line 1235, in q_run
test_suite = inq.get()
File "/usr/lib/python2.6/multiprocessing/queues.py", line 89, in get
self._rlock.acquire()
KeyboardInterrupt
test_suite = inq.get()
File "/usr/lib/python2.6/multiprocessing/queues.py", line 89, in get
self._rlock.acquire()
KeyboardInterrupt
File "/usr/lib/python2.6/multiprocessing/queues.py", line 89, in get
Traceback (most recent call last):
* when i cancelled the above test run, I also noticed some empty
directories hung around (three to be precise, in my case named
/tmp/tmpFyGS6Z, /tmp/tmpJG0SnH, /tmp/tmpjFImyq. If it helps,
all three had an owner group of noaccess:noaccess, so I suspect
a bug with cleanup when dealing with tests that use su_wrap.
Huh. I'd say it's less a bug w/ the cleanup, and more a bug that those
tests are creating files outside of the test directories. I'll see what
I can do about this.
* I then ran the tests again with -j4 and ^C'd them after a few
minutes, and got a message about a traceback, but no actual
traceback:
That's just bizarre, but I don't really have more ideas than what I said
above.
# logging to /tmp/tmpklqfAK.pkg-test.log
^CAll spawned processes should be terminated, now cleaning up
directories.
Directories successfully removed.
Traceback (most recent call last):
-Shawn
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
