On Jan 24, 2008 10:12 PM, Dan Fabulich <[EMAIL PROTECTED]> wrote: >
> Most of the JUnit reporting tools, including Ant's JunitReport task and > Maven's surefire-report plugin (but also including Hudson and others) > incorrectly assume that all of those testcases are methods of MyTestSuite. > As a result, it makes it look like all of your tests are methods of one > class in one package. If you've got two "testOutput" methods, they'll be > indistinguishable (except by stacktrace, if one is present). > > The same thing happens when you run TestNG tests: it always generates all > of the test results in one XML file. (Though it contains more > information, see below.) > > More generally, both Ant's JunitReport and surefire-report incorrectly > assume a one-to-one relationship between TestCase classes, XML files, and > "suites" of tests. They assume that every XML file contains one suite of > tests, which is the same thing as a TestCase class, and all of the tests > in the suite are just methods of the TestCase class. This assumption is > wrong. I guess the reason for the assumption is that nobody normally writes a test suite for JUnit, as it is needless coding that a pattern like **/*Test.class could do at the build file level, where you can also turn different tests on and off. But as you point out, it doesnt hold for other test systems. > Ant 1.6 (I think?) introduced a helpful attribute on all of its <testcase> > classes to include the class name of the class on every <testcase> > element: > > Example 7 > <testsuite name="com.mycompany.MyTestSuite" failures="0" errors="0" > tests="4" time="0.031"> > <properties><!-- ... --></properties > <testcase name="test1a" time="0.0" classname="com.mycompany.Test1" /> > <testcase name="test1b" time="0.0" classname="com.mycompany.Test1" /> > <testcase name="test2a" time="0.0" classname="com.mycompany.Test2" /> > <testcase name="test2b" time="0.0" classname="com.mycompany.Test2" /> > <testcase name="test3a" time="0.0" classname="com.mycompany.Test3" /> > <testcase name="test3b" time="0.0" classname="com.mycompany.Test3" /> > </testsuite> I forget when, but take your word for it. > This is good, because it allows you to figure out which class the method > really belongs to. This is what TestNG outputs when it generates > JUnit-like results. Unfortunately, nobody honors the "classname" > attribute, including Ant 1.7's JunitReport task! You contributions to the XSL file are welcome :) > > Even if we did try to honor that information and report on it in the HTML, > how would we do it? Right now, the reports are organized in terms of > packages, and within the packages you'll get a list of classes (assumed > incorrectly to be the same thing as "suites"), and within the list of > classes you'll get a list of test method names. There's no room for > "suites" (as distinct from classes) in these reports at all. > > And arguably there shouldn't be a place for this information, because > apparently 80-90% of the time suites aren't being used; adding a "suites" > section would be redundant in those cases. > > We could throw away the "suite" wrapper and pretend as if the tests were > just classes and methods, but note something else: the sum of the "time" > attribute on the <testcase> elements is normally different from the "time" > attribute on the <testsuite>. That's because the timer has a 16ms > resolution on most operating systems. > > In bug SUREFIRE-433, it was argued that we should generate separate JUnit > XML files for every class. You could even imagine converting the results > from Example 7 into results that looked like Example 2, throwing away all > of the suite-level information. > > I certainly want to be conservative in what I emit and liberal in what I > accept, but I *really* don't want to throw away information, especially at > the XML level. With that said, as we're trying to make the report look > something like the way it looks today, we're going to lose suite-level > information in the HTML. > > What I propose is that, in order to avoid destroying information, Surefire > should generate XML that looks like Example 7 (all-in-one-file), and not > try to fake it to look like Example 2 (one-file-per-class). (TestNG's > junit-like output also generates files like Example 7.) However, when it > comes time to generate an HTML report, surefire-reports will discard > suite-level information, and treat large suites like Example 7 as if they > had been presented in separate files like Example 2. I'd argue that all > of the other JunitReport-like tools (including Ant) should probably follow > the same lead. I think that if everyone else has a bug, its hard to call it a bug. More a presentation choice :) > My proposal is not obviously right, because, again, most other JUnit > report tools also have this reporting bug; when they try to format TestNG > output or Surefire output, they'll incorrectly report all methods to be > members of the suite class. Maybe since we expect EVERYONE to have this > bug, we should be even more conservative in what we emit and generate > multi-file output, just so the other tools will know how to interpret it > correctly. That would make it more likely that the HTML output would > contain complete information. > > What do other people think? Agree, disagree? I do need to allocate time to fix reporting in Ant, both what we generate (assuming a junit4 task), and what the tool processes. Me, I'm more concerned with the problem of trying to merge log results from a server and a client, as the current reports only pick the stdout and stderr from one side of the party, -steve
