On 04/28/2014 09:31 PM, John Ferlan wrote:
Just figured I'd share... Over the last few months I periodically run every single 'tp-libvirt' virsh test against recent upstream changes (libvirt, virt-test, and tp-libvirt) mostly as a way to ensure recent changes don't cause regressions. During this time - I've found a few libvirt upstream issues and a few test issues prior to release checkpoints - so all in all a good use of CPU cycles. A run such as this takes a full day to complete - it's usually a weekend type run.
Awesome! It is good to have your help to keep the libvirt tests stable, and I'm even happier to see the tests being useful to find regressions in the software.
My most recent run (changes as of Fri 4/25 @ 10PM) had the following results: TOTAL: 3174 PASS : 2889 FAIL : 17 SKIP : 268 PCT : 99.4% (PASS/(TOTAL-SKIP)) I ran using "./run -t libvirt --test virsh --no virsh.dompmsuspend". I chose to not include 'dompmsuspend' since I believe there's an upstream change that has caused the test to just wait forever for something that won't happen. I haven't had the cycles to dig into the condition, but it was running at some time in the recent past and I do remember seeing changes go by upstream in this space - I just wasn't paying close attention to them and have to find them again. Of the 17 failures - 4 are fixed by a patch pushed today (and validated) - 2 are "known" failures w/ a bz. - 2 that I know are directly related to a recent upstream fix
The numbers are quite impressive, and a testament to the great work the team taking care of the libvirt tests is doing!
I believe the rest have differences due to some upstream change in behaviour that's not reflected in the test. More than likely a fix to some problem made after the test was created - it's just a matter of time to investigate and determine where the change was made in order to make the correct change in the test to reflect the change. My environment does have 6 'tp-libvirt' changes that are waiting for review and merge.
My time to chime in and help out with the reviews is very limited these days, I'll try to at least help with some of your fixes, though.
Cheers, Lucas _______________________________________________ Virt-test-devel mailing list [email protected] https://www.redhat.com/mailman/listinfo/virt-test-devel
