I'm not sure. Maybe we could offer some kind of opt-in or opt-out for this feature (jenkins plugin?). Maybe having an opt-out is cheaper than validating the feature on a good number of CI tools.
My gut feel tells me that there are tools that won't support this new format. Cheers! On Fri, Apr 19, 2013 at 4:55 PM, Luke Daley <luke.da...@gradleware.com>wrote: > > On 19/04/2013, at 3:34 PM, Szczepan Faber <szczepan.fa...@gradleware.com> > wrote: > > > The schema for this file used to be somewhere in the web (I've found it > some time ago when I needed it). However, it's probably more important to > be compatible with most consumers (e.g. all/most CI servers). > > The fact that there is a schema doesn't really change anything, because it > appears that no one respects it. > > > I think we should be very careful changing the format of the junit xml > reports (e.g. test out integrations with other CIs, etc). > > > > I think it would be very useful if Gradle supported output per test. > > The only way to do that (that we've identified) is to change the XML file, > so this statement appears in contention with your previous. > > Are you saying we should change it, but be careful in doing so? Or should > not change it and need to find another way? > > > > > Cheers! > > > > > > On Fri, Apr 19, 2013 at 2:33 PM, Luke Daley <luke.da...@gradleware.com> > wrote: > > Hi all, > > > > In the “JUnit XML” file that we currently produce we do not associate > output (std, err) with individual test cases, but with the suite (in > practice: class) as a whole. This is causing problems in integrating with > https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Attachments+Plugin. > > > > Given: > > > > package junit; > > > > import org.junit.Test; > > import org.junit.runner.RunWith; > > import org.junit.runners.JUnit4; > > > > @RunWith(JUnit4.class) > > public class TheTest { > > > > @Test > > public void t() { > > System.out.println("foo"); > > } > > > > } > > > > We generate XML like: > > > > <?xml version="1.1" encoding="UTF-8"?> > > <testsuite name="junit.TheTest" tests="1" failures="0" errors="0" > timestamp="1970-01-01T00:00:00" hostname="hasdrubal.local" > time="1.366372749405E9"> > > <properties/> > > <testcase name="t" classname="junit.TheTest" time="0.001"/> > > <system-out><![CDATA[foo > > ]]></system-out> > > <system-err><![CDATA[]]></system-err> > > </testsuite> > > > > After digging through the Jenkins code for how it parses these files, I > discovered that it supports: > > > > <?xml version="1.1" encoding="UTF-8"?> > > <testsuite name="junit.TheTest" tests="1" failures="0" errors="0" > timestamp="1970-01-01T00:00:00" hostname="hasdrubal.local" > time="1.366372749405E9"> > > <properties/> > > <testcase name="t" classname="junit.TheTest" time="0.001"> > > <system-out><![CDATA[foo > > ]]></system-out> > > <system-err><![CDATA[]]></system-err> > > </testcase> > > </testsuite> > > > > That is, you can associate output with an individual testcase. > > > > I did some experiments with Maven to see what it does. Maven doesn't > record any stdio information in the XML file usually; it puts text files on > the side. However, I happened upon a case where it does. If you try and > execute a non public class you get: > > > > <?xml version="1.0" encoding="UTF-8" ?> > > <testsuite failures="0" time="0.006" errors="2" skipped="0" tests="2" > name="junit.TheTest"> > > > > <testcase time="0.002" classname="junit.TheTest" > name="initializationError"> > > <error message="Test class should have exactly one public > constructor" type="java.lang.Exception">java.lang.Exception: Test class > should have exactly one public constructor > > at > org.junit.runners.BlockJUnit4ClassRunner.validateOnlyOneConstructor(BlockJUnit4ClassRunner.java:131) > > … > > </error> > > <system-out>@SLTests run: 3, Failures: 0, Errors: 0, Skipped: 0, > Time elapsed: 0.033 sec > > @SLRunning junit.TheTest > > </system-out> > > </testcase> > > <testcase time="0.001" classname="junit.TheTest" > name="initializationError"> > > <error message="Class junit.TheTest should be public" > type="java.lang.Exception">java.lang.Exception: Class junit.TheTest should > be public > > at > org.junit.runners.model.FrameworkMethod.validatePublicVoid(FrameworkMethod.java:87) > > … > > </error> > > </testcase> > > </testsuite> > > > > The <system-out> elements are nested under <testcase> there. So it seems > there is some precedent for this. > > > > > > I have some incentive to at least get this working for a client in > Jenkins. Where "get it working" would be to have the <system-out> nested > under <testcase> so it can do the correct association. > > > > I have no idea who else supports this format though. It's hard to > predict given that there is no schema for this XML file. > > > > -- > > Luke Daley > > Principal Engineer, Gradleware > > http://gradleware.com > > > > Join me at the Gradle Summit 2013, June 13th and 14th in Santa Clara, > CA: http://www.gradlesummit.com > > > > > > --------------------------------------------------------------------- > > To unsubscribe from this list, please visit: > > > > http://xircles.codehaus.org/manage_email > > > > > > > > > > > > -- > > Szczepan Faber > > Principal engineer@gradleware; Lead@mockito > > Join me at the Gradle Summit 2013, June 13th and 14th in Santa Clara, > CA: http://www.gradlesummit.com > > -- > Luke Daley > Principal Engineer, Gradleware > http://gradleware.com > > Join me at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: > http://www.gradlesummit.com > > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > > -- Szczepan Faber Principal engineer@gradleware; Lead@mockito Join me at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: http://www.gradlesummit.com
