Just another case... Functionally testing ajax heavy sites (with Geb of course), or any concurrent type code. With the ajax case, I find that I don't have much faith the *tests* are good until I have run them a few times to get the timeouts right.
On 27/12/2010, at 6:47 AM, Adam Murdoch <[email protected]> wrote: > > On 24/12/2010, at 6:56 AM, Peter Niederwieser wrote: > >> >> >> Adam Murdoch-3 wrote: >>> >>>> Sometimes there are external factors. >>> >>> I guess I was after something a bit more concrete. >>> >> >> A few examples that come to my mind: >> >> - A test that generates random inputs (property-based testing, testing for >> deadlocks, etc.) >> - A test that reads inputs/outputs from an external Excel sheet >> - An integration test hitting a remote (test) database >> - An acceptance test that runs against a live website >> > > I like these examples. It feels like there are a few aspects here: > > * Tests which use resources other than those on the classpath, such as the > excel sheet, database, or web site. > > At the moment, we model these as inputs and outputs of the test task. And > only for local files. But it might be interesting to model other sorts of > resources - such as remote files, or database instances or web applications. > Then, we can skip a given test if the resources it uses have not changed > since last time the test was executed. > > Not sure how useful this is for incremental testing. But it is useful for > things such as test setup and tear down, where Gradle can make sure the > resources are deployed before the test and cleaned up after the test. And for > reusing the tests in different contexts. A test can declare a dependency on a > particular web app, and Gradle can inject the right config into the test: an > embedded deployment for a dev build, a test deployment in staging for the ci > build, and the production web app as a smoke test for the deployment build. > > > * Tests which have some non-deterministic element > > Each test execution is a data point that increases confidence in the system > under test. In a sense, all tests are like this to some degree. > > To me, running the test task multiple times from the command-line to increase > confidence is in itself a test case, and is probably worth capturing in the > automated test suite somehow. > > One option is to define this in the build: we might introduce the concept of > a test suite, and allow you to specify things such as how many times a given > test or suite should be executed for us to have confidence that the test has > 'passed'. We might add other constraints too: this test must run repeatedly > for 8 hours, this test must run on a machine with at least 2 physical cores, > this test must run repeatedly at the rate of 30/minute, this test must run > concurrently on at least 4 difference machines, etc. > > These constraints would be useful for other use cases to, such as: this test > must run under java 5, this test must run on a linux machine, this test must > run on each of a windows, linux and mac machine, this test must run on a > machine with oracle installed, this test must run against each of oracle, > postgres, mysql, etc. > > Regardless, I think it is an important point you make that tests are not > always deterministic and may need to execute multiple times. We should > capture this some how. > > > * Tests which run at multiple points in the application lifecycle. > > For example, some acceptance tests which run in the developer pre-commit > build, the ci build, and the deployment build. You might inject a different > web app at each stage, and you might add difference constraints at each stage. > > I wonder if you might also want different up-to-date strategies at each > stage. For a dev build, I don't want to run a set of acceptance tests if I > haven't changed the system under test since I last ran them (and they've > passed according to whatever constraints have been specified). But, for a > deployment build, I might want to specify that the acceptance tests must > always be run. > > > -- > Adam Murdoch > Gradle Developer > http://www.gradle.org > CTO, Gradle Inc. - Gradle Training, Support, Consulting > http://www.gradle.biz >
