Benjamin Bentmann wrote:
I see. Now, if the user can provide an option to Surefire, indicating
that the unit tests do not have inter-class dependencies, wouldn't this
allow to get back the old behavior? I.e. enabling Surefire to pass test
classes individually into TestNG?
That option sounds really confusing. What would you call it? How would
you document it? You'd have to know a lot about the guts of Surefire and
TestNG for an option like that to make much sense.
Not to mention that it'd be error prone. Developer X would assess the
tests, verifying that there are no inter-class dependencies, and turn on
the flag. Later, developer Y could introduce an inter-class dependency
and go to run the tests, but find that it's not working. It'd be hard to
figure out why without knowing a lot about how Surefire is doing its job.
More generally, I don't think class-level reporting really makes sense in
TestNG as it's designed today. Right now, TestNG provides listeners with
the following notifications:
onStart
onTestStart
onTestSuccess
onTestFailure
onTestSkipped
onTestFailedButWithinSuccessPercentage
onConfigurationSuccess
onConfigurationFailure
onConfigurationSkip
onFinish
Note that there's nothing here about "onClassStart/Finish" or even
"onGroupStart/Finish".
If TestNG exposed a listener, we could easily attach to it, but I see that
as a feature of TestNG. We could certainly try to write TestNG features
in Surefire, but the TestNG code knows best whether, for example, your
classes have any inter-dependencies, when your classes start and stop, and
more generally how to construct a full test run comprised of suites,
groups, classes and tests.
Surefire shouldn't be in the business of reimplementing that logic,
especially since we expect this feature to be generally useful. It should
be a part of TestNG, if it matters at all.
I would not have any problems with a single file per method. Indeed, I think
I like the idea: Test methods are the units that succeed or fail, so if test
method TestClass.testFoo() fails, I would investigate
TestClass.testFoo-output.txt and can easily concentrate on its output
without any need to skip over the output from all the other successful
methods. I think that would be worth another option ;-)
"TestNG is designed to cover all categories of tests: unit, functional,
end-to-end, integration, etc..."
In TestNG, especially when one method @dependsOn another method, the test
failures aren't units, and aren't even meant to be units. (This can be a
huge benefit for slow integration tests, where earlier tests leave behind
fixtures that later tests are meant to consume.)
If all you're doing is unit tests, you may prefer to use JUnit 4. :-)
Anyway, in thinking about it a little harder I realized that we can't even
necessarily split the stdout into one file per method, since the tests may
be run in parallel!
I'm pretty sure we can do that logging thing, at least. :-p
-Dan
