I spent some time solving a thorny problem over the last couple of days;
I'm not 100% sure I like the solution to it, so I wanted to write a bit
about it to see what other people think.
The problem occurs in cases where one test suite incorporates tests from
multiple test classes. This is the normal use case for a JUnit TestSuite.
You'll typically write a JUnit suite like this:
Example 1
public class MyTestSuite extends TestSuite {
public static Test suite() {
MyTestSuite suite = new MyTestSuite();
suite.addTestSuite(FooTestCase.class);
suite.addTest(BarTestSuite.suite());
return suite;
}
}
(In TestNG, suites are defined using XML files, and are considerably more
common in TestNG-land.)
The question I'm trying to answer is: how should the results of running a
TestSuite be represented in XML, and how should that XML be represented in
HTML? You might think this would be a solved problem, but it doesn't
appear to be. :-( Bugs SUREFIRE-433 and TESTNG-204 all relate to this, as
well as Ant bug 24106.
http://jira.codehaus.org/browse/SUREFIRE-433
http://jira.opensymphony.com/browse/TESTNG-204
http://issues.apache.org/bugzilla/show_bug.cgi?id=24106
First, it seems like most people don't even use test suites, preferring to
simply run every test in their src/test/java directory. If you do this,
you get a bunch of XML files, one file per TestCase class, like this:
Example 2
TEST-com.mycompany.Test1.xml
TEST-com.mycompany.Test2.xml
TEST-com.mycompany.Test3.xml
The individual XML files look like this:
Example 3
<testsuite name="com.mycompany.Test1" failures="1" errors="0" tests="2"
time="0.031">
<properties><!-- ... --></properties
<testcase name="test1a" time="0.0" />
<testcase name="test1b" time="0.0">
<failure type="java.lang.AssertionError" message="I don't like this
movie!">java.lang.AssertionError: I don't like this movie!
at org.testng.Assert.fail(Assert.java:84)
at org.testng.Assert.fail(Assert.java:91)
at com.mycompany.Test1.test1b(Test1.java:14)
</failure>
</testcase>
</testsuite>
That's all fine, but what if you want to use a test suite? When you do
that in Ant or in Maven today, you get just one XML file:
Example 4
TEST-com.mycompany.MyTestSuite.xml
In Surefire 2.4, the XML unfortunately looks like this:
Example 5
<testsuite name="com.mycompany.MyTestSuite" failures="0" errors="0" tests="4"
time="0.031">
<properties><!-- ... --></properties
<testcase name="test1a" time="0.0" />
<testcase name="test1b" time="0.0" />
<testcase name="test2a" time="0.0" />
<testcase name="test2b" time="0.0" />
<testcase name="test3a" time="0.0" />
<testcase name="test3b" time="0.0" />
</testsuite>
Note that information has been lost here, and the information that *is*
provided is misleading. Specifically, we lost the fact that test1a was a
method of com.mycompany.Test1; all we know is its method name.
Furthermore, the report misleadingly seems to suggest that test1a is a
method of MyTestSuite, whereas in fact MyTestSuite just has a suite()
method and no test methods of its own.
Most of the JUnit reporting tools, including Ant's JunitReport task and
Maven's surefire-report plugin (but also including Hudson and others)
incorrectly assume that all of those testcases are methods of MyTestSuite.
As a result, it makes it look like all of your tests are methods of one
class in one package. If you've got two "testOutput" methods, they'll be
indistinguishable (except by stacktrace, if one is present).
The same thing happens when you run TestNG tests: it always generates all
of the test results in one XML file. (Though it contains more
information, see below.)
More generally, both Ant's JunitReport and surefire-report incorrectly
assume a one-to-one relationship between TestCase classes, XML files, and
"suites" of tests. They assume that every XML file contains one suite of
tests, which is the same thing as a TestCase class, and all of the tests
in the suite are just methods of the TestCase class. This assumption is
wrong.
Ant 1.6 (I think?) introduced a helpful attribute on all of its <testcase>
classes to include the class name of the class on every <testcase>
element:
Example 7
<testsuite name="com.mycompany.MyTestSuite" failures="0" errors="0" tests="4"
time="0.031">
<properties><!-- ... --></properties
<testcase name="test1a" time="0.0" classname="com.mycompany.Test1" />
<testcase name="test1b" time="0.0" classname="com.mycompany.Test1" />
<testcase name="test2a" time="0.0" classname="com.mycompany.Test2" />
<testcase name="test2b" time="0.0" classname="com.mycompany.Test2" />
<testcase name="test3a" time="0.0" classname="com.mycompany.Test3" />
<testcase name="test3b" time="0.0" classname="com.mycompany.Test3" />
</testsuite>
This is good, because it allows you to figure out which class the method
really belongs to. This is what TestNG outputs when it generates
JUnit-like results. Unfortunately, nobody honors the "classname"
attribute, including Ant 1.7's JunitReport task!
Even if we did try to honor that information and report on it in the HTML,
how would we do it? Right now, the reports are organized in terms of
packages, and within the packages you'll get a list of classes (assumed
incorrectly to be the same thing as "suites"), and within the list of
classes you'll get a list of test method names. There's no room for
"suites" (as distinct from classes) in these reports at all.
And arguably there shouldn't be a place for this information, because
apparently 80-90% of the time suites aren't being used; adding a "suites"
section would be redundant in those cases.
We could throw away the "suite" wrapper and pretend as if the tests were
just classes and methods, but note something else: the sum of the "time"
attribute on the <testcase> elements is normally different from the "time"
attribute on the <testsuite>. That's because the timer has a 16ms
resolution on most operating systems. By starting the timer at the
beginning of the entire suite, we can capture overall time that we lose
when we turn the timer on and off. (And what if the tests had been run in
parallel?)
In bug SUREFIRE-433, it was argued that we should generate separate JUnit
XML files for every class. You could even imagine converting the results
from Example 7 into results that looked like Example 2, throwing away all
of the suite-level information.
I certainly want to be conservative in what I emit and liberal in what I
accept, but I *really* don't want to throw away information, especially at
the XML level. With that said, as we're trying to make the report look
something like the way it looks today, we're going to lose suite-level
information in the HTML.
What I propose is that, in order to avoid destroying information, Surefire
should generate XML that looks like Example 7 (all-in-one-file), and not
try to fake it to look like Example 2 (one-file-per-class). (TestNG's
junit-like output also generates files like Example 7.) However, when it
comes time to generate an HTML report, surefire-reports will discard
suite-level information, and treat large suites like Example 7 as if they
had been presented in separate files like Example 2. I'd argue that all
of the other JunitReport-like tools (including Ant) should probably follow
the same lead.
My proposal is not obviously right, because, again, most other JUnit
report tools also have this reporting bug; when they try to format TestNG
output or Surefire output, they'll incorrectly report all methods to be
members of the suite class. Maybe since we expect EVERYONE to have this
bug, we should be even more conservative in what we emit and generate
multi-file output, just so the other tools will know how to interpret it
correctly. That would make it more likely that the HTML output would
contain complete information.
What do other people think? Agree, disagree?
-Dan
