I would like to propose that we *tag* tests as small, medium, or large. Small tests are measured in millisecionds, medium are measured in seconds up to a minute, and large are measured in minutes or hours
We should run small tests locally, all the time. Medium tests should gate commits. Large tests should be run on a periodic basis, hourly, daily our whatever is required to catch errors on a regular basis. I think we have mechanisms to run all three sets of tests. The CI folks have stuff to create environments and /do things/, we have local test running, and we have both commit level testing, and periodic CI tests. What we don't yet have is clear definition on where various tests belong, and how to add them. Nor do we have a clear way to mark tests in the unit test suite as "medium" so that developers can have a very fast set of tests to run, and we don't have so much backpressure on adding "more expensive" tests. We've been working on adding a backup method to the API client. One test I wanted to add was a functional test that did no patching anywhere. Such an end-to-end test would give us more confidence in the API method. We currently have something like that for backup/restore, but only as a CI test running on jenkins. As I understand it, there are two roadblocks to getting such a functional test in our test suite: 1. such a test would drastically slow down our other tests; 2. we don't have an easy way to set up a clean full environment against which to run the test, Solving the first should be as simple as only running the test when some env variable is set (e.g. JUJU_TESTING_INCLUDE_SLOW). However, solving the second roadblock isn't so trivial. Ideally the test would create an LXC into which it would bootstrap. Then the test would set up the environment however it needed before running the API client method against that environment. I imagine that this could be addressed with something similar to a local provider but where the master is also in an LXC (and the kinks with that are worked out). >From what I understand there has been some discussion on this already. How hard would it be to do this? While my interest is specifically for backup/restore, I imagine that other API client methods (and perhaps other parts of juju) would benefit from the above functionality. Thoughts? -eric -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
-- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
