We should start with defining which kinds of criteria we’re going to support when a user requests that some subset of tests should be run. Then we can bike shed the format. There are at least 4 options we might choose from:
1. The test's implementation class and method. That is, run only those tests whose implementation matches the requested method, everywhere it might appear in the test hierarchy. There are some downsides to using this that might not be immediately apparent: - Not every test framework uses methods from jvm classes to implement tests. It might use some spec language, for example. - Doesn’t work when I’m testing something that isn’t jvm based, such as native code or javascript. - There isn’t necessarily a 1-to-1 relationship between a test and its implementation method. For example, if I’m using a custom JUnit test runner or parameterised tests. - It’s not always possible to identify the implementation method of a test. 2. The position of the test in the test runtime hierarchy. The test hierarchy is how tests are organised at runtime into composite tests and atomic tests. This is the most flexible option. 3. The source file that the test is implemented in. This is what you can do now with -Dtest.single. To select individual tests, you also need to give one of the above (implementation method name or atomic test name) to select the tests from the matching source files. 4. Some framework specific namespace. For example, using a Junit category or TestNG group. Option #3 is pretty awful. Any or all of the others make sense. On 18 Nov 2013, at 10:13 pm, Szczepan Faber <szczepan.fa...@gradleware.com> wrote: > Hey, > > I'll come up with an initial spec for this, please give feedback. The spec > lives here: > https://github.com/gradle/gradle/blob/master/design-docs/test-selection.md > I'm including the content here because some formatting is not good. > > Use case: legacy integration tests are slow, need a way to execute a single > test method, from command line. > > Options: > > 1. Change the semantics of test.include so that it supports syntax like > {ant-file-pattern}#{testMethodName}, e.g. **/FooTest#someMethod > Test implements PatternFilterable, which declares the exact behavior of > include / exclude. In order to implement this option, we would need to > introduce a mildly breaking change and make Test no longer extend > PatternFilterable. It is kind of a mildly breaking because we would keep all > the existing public methods in the Test type. If users extends Test task, > they would probably need to recompile. For command line, we introduce > --include option for the Test task. Pros: api remains small, Cons: breaking > change. > > 2. Introduce new dsl (various ideas lumped together): > > test { > selection { > //using 'include' wording here could make it confusing with test.include > select "**/Foo*#someOtherMethod", "**/Foo*#someMethod" > selections = [] > > //in/out > in "**/Foo*#someOtherMethod", "**/Foo*#someMethod" > out "**/Foo*#someOtherMethod", "**/Foo*#someMethod" > > //if we deprecate test.include, we could do > include "**/Foo*#someOtherMethod", "**/Foo*#someMethod" > includes = [] > > //keep method selection separate > includeMethod > includeMethods = [] > > include { > method > methods = [] > } > > //some other elements we could add in future > exclude > excludes = [] > > unselect > unselections = [] > > include { > descendantsOf 'com.foo.SomeBaseClass' > annotatedWith 'com.foo.Slow' > matching { descriptor, testClass -> > //... > } > } > } > } > > Then add consistent commandline support, e.g. > > gradle test --select **/Foo.java#someTest > > 3. It would be good to consider the command line interface when designing the > dsl because they need to be consistent. The convenient command line should > support: > - selecting class + method in one option > - allow wildcards for classes (wildcards for methods are not practical > to implement) > - support both, file separators '/' or fqn separators '.' > > --select com/bar/**/Foo.java#someTest > --select Foo#someTest > --select com.foo.bar.Foo#otherTest > > 4. Here's api I was thinking about as a first story, please give feedback: > > test { > //good old include > include '**/*SomeIntegrationTest' > > //new stuff > selection { > include { > method 'method1', 'method2' > methods = [] > } > } > } > > Plus, a convenience method directly in the test task, so that all above could > be inlined into: > > test { > select '**/*SomeIntegrationTest#method1,method2' > //or: > select '**/*SomeIntegrationTest#method1', '**/*SomeIntegrationTest#method2' > } > > Then, command line support could be: > > gradle test --select **/*SomeIntegrationTest#method1,method2 > > I would also deprecate test.single property > > Cheers! > > > On Fri, Nov 15, 2013 at 4:23 AM, Adam Murdoch <adam.murd...@gradleware.com> > wrote: > > On 13 Nov 2013, at 7:36 am, Szczepan Faber <szczepan.fa...@gradleware.com> > wrote: > >> Heya, >> >> I'm keen on solving this use case: developer in a large app has tons of long >> running tests. He wants to run only a *single* test from the command line >> (e.g. we currently support *all* tests from a *single* test class via >> test.single). >> >> There's also a pull request: https://github.com/gradle/gradle/pull/193 that >> shows that the community is also keen on getting it sorted out. >> >> How do we want approach this? >> >> 1. Make the test.includes / test.excludes more robust and support some kind >> of notation like "SomeTest#someMethod". This is kind of awkward because Test >> extends PatternFilterable that clearly defines what are includes and >> excludes (they are not test methods, they are ant file patterns). On the >> plus side, this would work with our existing means of specifying >> includes/excludes (existing DSL, existing test.single property) >> >> 2. Add some new API for this, for example: >> >> test.selection.include ... >> >> It might be good to put the test selection api behind some new DSL >> layer/object (e.g. 'selection') because it can potentially grow: >> include/exclude tests by some custom criteria, e.g. test class hierarchy, >> etc. BTW. for the latter we have use cases in our own codebase (cross >> version tests now are picked up by naming convention but it would be nicer >> if they were picked up by parent class, etc.). >> >> If we go down this path, we need to add command line support for it and >> perhaps consider deprecation of the existing include / exclude. >> >> Do we have some other options? Thoughts? > > I’d go with option #2. The current includes and excludes are really intended > to specify where to look for tests. Option 2 adds another filter over this to > specify some criteria to use for selecting tests (beyond where they > physically live). > > I’d also add command-line support and deprecate the system property. > > > -- > Adam Murdoch > Gradle Co-founder > http://www.gradle.org > VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting > http://www.gradleware.com > > > > > > > -- > Szczepan Faber > Principal engineer@gradle; Founder@mockito -- Adam Murdoch Gradle Co-founder http://www.gradle.org VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com
