The longer-term plan is to deliver the test events to the CI server via the tooling API, and we can make these events as rich as we like. Doesn't help us now, though.
There are 3 case we want to support here: 1. Don't generate the XML at all. For example, don't generate the xml if nothing went wrong, or if I'm doing a dev build, or if I'm using a CI server that doesn't use it (TeamCity uses the events already, Bamboo doesn't show the output). 2. Generate the XML with output per atomic test. 3. Generate the XML with output per test class (aka 'broken mode'). We want to deprecate and remove #3 as soon as is practical, once #2 is available. The question here is what we would break if we just switched to #2? We should put some effort into answering that question. On 19/04/2013, at 10:33 PM, Luke Daley <luke.da...@gradleware.com> wrote: > Hi all, > > In the “JUnit XML” file that we currently produce we do not associate output > (std, err) with individual test cases, but with the suite (in practice: > class) as a whole. This is causing problems in integrating with > https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Attachments+Plugin. > > Given: > > package junit; > > import org.junit.Test; > import org.junit.runner.RunWith; > import org.junit.runners.JUnit4; > > @RunWith(JUnit4.class) > public class TheTest { > > @Test > public void t() { > System.out.println("foo"); > } > > } > > We generate XML like: > > <?xml version="1.1" encoding="UTF-8"?> > <testsuite name="junit.TheTest" tests="1" failures="0" errors="0" > timestamp="1970-01-01T00:00:00" hostname="hasdrubal.local" > time="1.366372749405E9"> > <properties/> > <testcase name="t" classname="junit.TheTest" time="0.001"/> > <system-out><![CDATA[foo > ]]></system-out> > <system-err><![CDATA[]]></system-err> > </testsuite> > > After digging through the Jenkins code for how it parses these files, I > discovered that it supports: > > <?xml version="1.1" encoding="UTF-8"?> > <testsuite name="junit.TheTest" tests="1" failures="0" errors="0" > timestamp="1970-01-01T00:00:00" hostname="hasdrubal.local" > time="1.366372749405E9"> > <properties/> > <testcase name="t" classname="junit.TheTest" time="0.001"> > <system-out><![CDATA[foo > ]]></system-out> > <system-err><![CDATA[]]></system-err> > </testcase> > </testsuite> > > That is, you can associate output with an individual testcase. > > I did some experiments with Maven to see what it does. Maven doesn't record > any stdio information in the XML file usually; it puts text files on the > side. However, I happened upon a case where it does. If you try and execute a > non public class you get: > > <?xml version="1.0" encoding="UTF-8" ?> > <testsuite failures="0" time="0.006" errors="2" skipped="0" tests="2" > name="junit.TheTest"> > > <testcase time="0.002" classname="junit.TheTest" name="initializationError"> > <error message="Test class should have exactly one public constructor" > type="java.lang.Exception">java.lang.Exception: Test class should have > exactly one public constructor > at > org.junit.runners.BlockJUnit4ClassRunner.validateOnlyOneConstructor(BlockJUnit4ClassRunner.java:131) > … > </error> > <system-out>@SLTests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time > elapsed: 0.033 sec > @SLRunning junit.TheTest > </system-out> > </testcase> > <testcase time="0.001" classname="junit.TheTest" name="initializationError"> > <error message="Class junit.TheTest should be public" > type="java.lang.Exception">java.lang.Exception: Class junit.TheTest should be > public > at > org.junit.runners.model.FrameworkMethod.validatePublicVoid(FrameworkMethod.java:87) > … > </error> > </testcase> > </testsuite> > > The <system-out> elements are nested under <testcase> there. So it seems > there is some precedent for this. > > > I have some incentive to at least get this working for a client in Jenkins. > Where "get it working" would be to have the <system-out> nested under > <testcase> so it can do the correct association. > > I have no idea who else supports this format though. It's hard to predict > given that there is no schema for this XML file. > > -- > Luke Daley > Principal Engineer, Gradleware > http://gradleware.com > > Join me at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: > http://www.gradlesummit.com > > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > -- Adam Murdoch Gradle Co-founder http://www.gradle.org VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com Join us at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: http://www.gradlesummit.com
