I think I have an issue running multiple tests. I want to add a dep8 test I've written to puppet. It already has a test, so I'm adding a second one.
Initially my adt-virt-lxc driver didn't support the "revert" capability and I found that I couldn't run both tests in a single adt-run. adt-virt-null also appears to not support the "revert" capability. It looks to me that this is what auto-package-testing uses, so I think if I were to upload these tests then we'll end up with a test failure. What I should do to fix the tests is to declare the restriction "breaks-testbed". But then adt-run won't be able to use adt-virt-null to run the test any more. It'll skip the test. That causes a non-zero exit status, and from what I can see auto-package-testing will treat that as a full failure. I've since implemented "revert" and "revert-full-system" to the LXC driver, so that gives us full support now. But of course the Jenkins QA lab doesn't use that right now. Have I got this right? This seems important to me for the server use case, since just about every test that tests a daemon will break the testbed. So I think we need to be using a driver that supports "revert-full-system" in the QA lab if we want to have more than one test per "daemon" package. I think we should already be declaring "breaks-testbed" where the test messes with a daemon's configuration, but if we do this now I think we'll start failing everything. Thoughts? Should I file a bug that says that multiple "breaks-testbed" tests are not supported (whether they declare that or not)? And then there are a slew of tests which should be declaring "breaks-testbed" but do not, and we can't make them do so because that would cause us to unnecessarily fail tests. -- Ubuntu-quality mailing list Ubuntu-quality@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-quality