I'm going forward with #1 currently. I think it gives us a good starting point to grow it further. Also, that's what most users expect I bet. Most users think in implementation classes and methods and would be very happy to execute tests based on matching class(es) and matching method(s). The origin use case is 'ability to run a selected single test method from a selected class because all tests in this class are slow'.
The new feature will be incubating. We still should try hard to think it through and design it well. Yet, there will be room for improvement and changes once the feature is out, specifically when we receive more feedback. Cheers! On Tue, Nov 26, 2013 at 2:07 AM, Adam Murdoch <adam.murd...@gradleware.com>wrote: > > We should start with defining which kinds of criteria we’re going to > support when a user requests that some subset of tests should be run. Then > we can bike shed the format. There are at least 4 options we might choose > from: > > 1. The test's implementation class and method. That is, run only those > tests whose implementation matches the requested method, everywhere it > might appear in the test hierarchy. > > There are some downsides to using this that might not be immediately > apparent: > - Not every test framework uses methods from jvm classes to implement > tests. It might use some spec language, for example. > - Doesn’t work when I’m testing something that isn’t jvm based, such as > native code or javascript. > - There isn’t necessarily a 1-to-1 relationship between a test and its > implementation method. For example, if I’m using a custom JUnit test runner > or parameterised tests. > - It’s not always possible to identify the implementation method of a test. > > 2. The position of the test in the test runtime hierarchy. The test > hierarchy is how tests are organised at runtime into composite tests and > atomic tests. > > This is the most flexible option. > > 3. The source file that the test is implemented in. This is what you can > do now with -Dtest.single. To select individual tests, you also need to > give one of the above (implementation method name or atomic test name) to > select the tests from the matching source files. > > 4. Some framework specific namespace. For example, using a Junit category > or TestNG group. > > Option #3 is pretty awful. Any or all of the others make sense. > > On 18 Nov 2013, at 10:13 pm, Szczepan Faber <szczepan.fa...@gradleware.com> > wrote: > > Hey, > > I'll come up with an initial spec for this, please give feedback. The spec > lives here: > https://github.com/gradle/gradle/blob/master/design-docs/test-selection.mdI'm > including the content here because some formatting is not good. > > Use case: legacy integration tests are slow, need a way to execute a > single test method, from command line. > > Options: > > 1. Change the semantics of test.include so that it supports syntax like > {ant-file-pattern}#{testMethodName}, e.g. **/FooTest#someMethod > Test implements PatternFilterable, which declares the exact behavior of > include / exclude. In order to implement this option, we would need to > introduce a mildly breaking change and make Test no longer extend > PatternFilterable. It is kind of a mildly breaking because we would keep > all the existing public methods in the Test type. If users extends Test > task, they would probably need to recompile. For command line, we introduce > --include option for the Test task. Pros: api remains small, Cons: breaking > change. > > 2. Introduce new dsl (various ideas lumped together): > > test { > selection { > //using 'include' wording here could make it confusing with > test.include > select "**/Foo*#someOtherMethod", "**/Foo*#someMethod" > selections = [] > //in/out > in "**/Foo*#someOtherMethod", "**/Foo*#someMethod" > out "**/Foo*#someOtherMethod", "**/Foo*#someMethod" > > //if we deprecate test.include, we could do > include "**/Foo*#someOtherMethod", "**/Foo*#someMethod" > includes = [] > > //keep method selection separate > includeMethod > includeMethods = [] > > include { > method > methods = [] > } > > //some other elements we could add in future > exclude > excludes = [] > > unselect > unselections = [] > > include { > descendantsOf 'com.foo.SomeBaseClass' > annotatedWith 'com.foo.Slow' > matching { descriptor, testClass -> > //... > } > } > } > } > > Then add consistent commandline support, e.g. > > gradle test --select **/Foo.java#someTest > > 3. It would be good to consider the command line interface when designing > the dsl because they need to be consistent. The convenient command line > should support: > - selecting class + method in one option > - allow wildcards for classes (wildcards for methods are not practical to > implement) > - support both, file separators '/' or fqn separators '.' > > --select com/bar/**/Foo.java#someTest > --select Foo#someTest > --select com.foo.bar.Foo#otherTest > > 4. Here's api I was thinking about as a first story, please give feedback: > > test { > //good old include > include '**/*SomeIntegrationTest' > > //new stuff > selection { > include { > method 'method1', 'method2' > methods = [] > } > } > } > > Plus, a convenience method directly in the test task, so that all above > could be inlined into: > > test { > select '**/*SomeIntegrationTest#method1,method2' > //or: > select '**/*SomeIntegrationTest#method1', > '**/*SomeIntegrationTest#method2' > } > > Then, command line support could be: > > gradle test --select **/*SomeIntegrationTest#method1,method2 > > I would also deprecate test.single property > > Cheers! > > > On Fri, Nov 15, 2013 at 4:23 AM, Adam Murdoch <adam.murd...@gradleware.com > > wrote: > >> >> On 13 Nov 2013, at 7:36 am, Szczepan Faber <szczepan.fa...@gradleware.com> >> wrote: >> >> Heya, >> >> I'm keen on solving this use case: developer in a large app has tons of >> long running tests. He wants to run only a *single* test from the command >> line (e.g. we currently support *all* tests from a *single* test class via >> test.single). >> >> There's also a pull request: https://github.com/gradle/gradle/pull/193that >> shows that the community is also keen on getting it sorted out. >> >> How do we want approach this? >> >> 1. Make the test.includes / test.excludes more robust and support some >> kind of notation like "SomeTest#someMethod". This is kind of awkward >> because Test extends PatternFilterable that clearly defines what are >> includes and excludes (they are not test methods, they are ant file >> patterns). On the plus side, this would work with our existing means of >> specifying includes/excludes (existing DSL, existing test.single property) >> >> 2. Add some new API for this, for example: >> >> test.selection.include ... >> >> It might be good to put the test selection api behind some new DSL >> layer/object (e.g. 'selection') because it can potentially grow: >> include/exclude tests by some custom criteria, e.g. test class hierarchy, >> etc. BTW. for the latter we have use cases in our own codebase (cross >> version tests now are picked up by naming convention but it would be nicer >> if they were picked up by parent class, etc.). >> >> If we go down this path, we need to add command line support for it and >> perhaps consider deprecation of the existing include / exclude. >> >> Do we have some other options? Thoughts? >> >> >> I’d go with option #2. The current includes and excludes are really >> intended to specify where to look for tests. Option 2 adds another filter >> over this to specify some criteria to use for selecting tests (beyond where >> they physically live). >> >> I’d also add command-line support and deprecate the system property. >> >> >> -- >> Adam Murdoch >> Gradle Co-founder >> http://www.gradle.org >> VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting >> http://www.gradleware.com >> >> >> >> > > > -- > Szczepan Faber > Principal engineer@gradle; Founder@mockito > > > > -- > Adam Murdoch > Gradle Co-founder > http://www.gradle.org > VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting > http://www.gradleware.com > > > > -- Szczepan Faber Principal engineer@gradle; Founder@mockito
