On 14/01/2013, at 9:06 AM, Matt Khan wrote: > Hi > > I made a start on a couple of design docs. > > This one is specifically about cucumber > https://github.com/3ll3d00d/gradle/blob/cuke-jvm/design-docs/support-cucumber-jvm.md > > This one is about opening up the Test task so that you don't have to have > someone commit to gradle core tasks when this sort of requirement comes up > https://github.com/3ll3d00d/gradle/blob/cuke-jvm/design-docs/extensible-test-frameworks.md > > In lieu of having an open api for adding test frameworks to the gradle test > task (as per the 2nd doc), I am inclined to suggest that the vast majority of > the benefit will come from improving gradle's test reporting rather than > baking cucumber-jvm into gradle core.
I'd like to do both these things. An API for adding a test framework means a consistent experience and user interface for running different types of tests in Gradle. Gradle can also fill in the missing bits that a test framework may not provide (parallel execution, progress reporting, logging, for example). Improving the test reporting means that reporting can be done on the native constructs of the test framework within that consistent user interface. Currently, a test framework needs to provide the following pieces: 1. A set of framework-specific options to expose to the DSL. 2. A test detector implementation that takes these options and the test classpath, and builds a sequence of test execution requests. 3. A test processor implementation that is instantiated in a worker process, and that takes a sequence of test execution requests, executes the tests as appropriate, and generates a sequence of test events as things execute. How do you think these pieces would look for cucumber? Does this structure even make sense for cucumber? > > My rationale is that the current gradle test reporting does not appear to > adequately describe what one can do with junit let alone cucumber. For > example, a suite is represented as the sum of the tests found within the > classes in the suite as opposed to a 1-n from suite to class to test. There are a couple of things going on here. Firstly, the structure of the report is heavily influenced by the information available in the Ant Junit XML results, which it used to be generated from. In 1.4, we've moved to using our own internal format. This was for performance reasons, but it also means that we are now able to rework how the report presents the results. The other aspect here is that there are multiple ways to aggregate the results: 1. By source structure. That is, the execution context is not considered, and the implementation class + method is used to aggregate the results. 2. By runtime structure. That is, the organisation of the tests at runtime into suites and so on is used to aggregate the results. The test report currently uses #1. This format is useful for those who don't care about why a test was executed, but instead care about what thing actually failed. Option #1 isn't going to work as we move into other test frameworks, and option #2 would work better. One question is whether we keep #1 in the report as well, for those (very common) cases where it does make sense to group by implementation class + method. > > One option is to extend the reporting to allow an arbitrary depth to the tree > (though probably 3-4 is as deep as it gets) as opposed to the current 2 level > structure & to include some other, ideally test centric, naming convention to > replace package/class/method. Absolutely. I think this is a good option. -- Adam Murdoch Gradle Co-founder http://www.gradle.org VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com
