On Wed, Oct 6, 2010 at 9:49 AM, Michael Nelson <[email protected]> wrote: > I just caused a test-fix using ec2 land with a windmill test that > *did* fail, but the run was reported as successful (details below). > > I noticed a thread back in June with the same subject, but am not sure > if it was resolved and this is a regression, or it was never > resolved... >
I never heard that it was resolved. > ---------- Forwarded message ---------- > From: Michael Nelson <[email protected]> > Date: Tue, Oct 5, 2010 at 9:13 PM > Subject: Test results: 652838-select-diffs-for-syncing => db-devel: SUCCESS > To: [email protected] > > > Tests started at approximately 2010-10-05 15:27:08.255466 > Source: > bzr+ssh://bazaar.launchpad.net/~michael.nelson/launchpad/652838-select-diffs-for-syncing > r9844 > Target: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel/ > r9862 > > 3745 tests run in 3:45:38.815244, 0 failures, 0 errors > This is the key bit. There are normally 10k+ tests, but the test runner has only detected ~4k. That means that a whole bunch of tests are being ignored. If you look in the attachment, you'll see there's a LayerIsolationError, and that every single failure or error after that is ignored. You can verify this by downloading the full output and using tools like "subunit-filter" or "testr load" to see it. Basically, subunit sees a "test:" command without a matching result command (e.g. "success:", "failure:") and then proceeds to assume that all of the rest of the input it gets is just regular stdout/stderr dumpage. That means there are two problems here: 1. zope.testrunner is generating invalid subunit output in the case of catastrophic layer failure. 2. subunit is not robust in the case of invalid input http://paste.ubuntu.com/507163/ shows a simplified stream that reproduces the problem, and I've uploaded https://code.edge.launchpad.net/~jml/subunit/masked-errors/+merge/37725 for the consideration of the other subunit hackers. I'll try to figure something out with zope.testrunner now. I'll report back to the list when I've made progress or given up. jml _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
