On 23/04/2013, at 11:00 PM, Adam Murdoch <adam.murd...@gradleware.com> wrote:
> > The longer-term plan is to deliver the test events to the CI server via the > tooling API, and we can make these events as rich as we like. Doesn't help us > now, though. > > There are 3 case we want to support here: > > 1. Don't generate the XML at all. For example, don't generate the xml if > nothing went wrong, or if I'm doing a dev build, or if I'm using a CI server > that doesn't use it (TeamCity uses the events already, Bamboo doesn't show > the output). > 2. Generate the XML with output per atomic test. > 3. Generate the XML with output per test class (aka 'broken mode'). > > We want to deprecate and remove #3 as soon as is practical, once #2 is > available. The question here is what we would break if we just switched to > #2? We should put some effort into answering that question. We may have to do #2 and #3 indefinitely depending on what the tools out there do. We need to work out the list of things we need to test. Here's what I can think of (rough priority order): 1. Jenkins 2. Hudson 3. Ant JUnit HTML reporter 4. Team City 5. Bamboo 6. Thoughtworks Go There are a lot of CI servers out there, we can't test them all. I'll try to get in touch with someone from the Jenkins team and ask them about why they support this format and what produces XML files in this format. If something that is in widespread use already does this, then that's a good sign. > > On 19/04/2013, at 10:33 PM, Luke Daley <luke.da...@gradleware.com> wrote: > >> Hi all, >> >> In the “JUnit XML” file that we currently produce we do not associate output >> (std, err) with individual test cases, but with the suite (in practice: >> class) as a whole. This is causing problems in integrating with >> https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Attachments+Plugin. >> >> Given: >> >> package junit; >> >> import org.junit.Test; >> import org.junit.runner.RunWith; >> import org.junit.runners.JUnit4; >> >> @RunWith(JUnit4.class) >> public class TheTest { >> >> @Test >> public void t() { >> System.out.println("foo"); >> } >> >> } >> >> We generate XML like: >> >> <?xml version="1.1" encoding="UTF-8"?> >> <testsuite name="junit.TheTest" tests="1" failures="0" errors="0" >> timestamp="1970-01-01T00:00:00" hostname="hasdrubal.local" >> time="1.366372749405E9"> >> <properties/> >> <testcase name="t" classname="junit.TheTest" time="0.001"/> >> <system-out><![CDATA[foo >> ]]></system-out> >> <system-err><![CDATA[]]></system-err> >> </testsuite> >> >> After digging through the Jenkins code for how it parses these files, I >> discovered that it supports: >> >> <?xml version="1.1" encoding="UTF-8"?> >> <testsuite name="junit.TheTest" tests="1" failures="0" errors="0" >> timestamp="1970-01-01T00:00:00" hostname="hasdrubal.local" >> time="1.366372749405E9"> >> <properties/> >> <testcase name="t" classname="junit.TheTest" time="0.001"> >> <system-out><![CDATA[foo >> ]]></system-out> >> <system-err><![CDATA[]]></system-err> >> </testcase> >> </testsuite> >> >> That is, you can associate output with an individual testcase. >> >> I did some experiments with Maven to see what it does. Maven doesn't record >> any stdio information in the XML file usually; it puts text files on the >> side. However, I happened upon a case where it does. If you try and execute >> a non public class you get: >> >> <?xml version="1.0" encoding="UTF-8" ?> >> <testsuite failures="0" time="0.006" errors="2" skipped="0" tests="2" >> name="junit.TheTest"> >> >> <testcase time="0.002" classname="junit.TheTest" name="initializationError"> >> <error message="Test class should have exactly one public constructor" >> type="java.lang.Exception">java.lang.Exception: Test class should have >> exactly one public constructor >> at >> org.junit.runners.BlockJUnit4ClassRunner.validateOnlyOneConstructor(BlockJUnit4ClassRunner.java:131) >> … >> </error> >> <system-out>@SLTests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time >> elapsed: 0.033 sec >> @SLRunning junit.TheTest >> </system-out> >> </testcase> >> <testcase time="0.001" classname="junit.TheTest" name="initializationError"> >> <error message="Class junit.TheTest should be public" >> type="java.lang.Exception">java.lang.Exception: Class junit.TheTest should >> be public >> at >> org.junit.runners.model.FrameworkMethod.validatePublicVoid(FrameworkMethod.java:87) >> … >> </error> >> </testcase> >> </testsuite> >> >> The <system-out> elements are nested under <testcase> there. So it seems >> there is some precedent for this. >> >> >> I have some incentive to at least get this working for a client in Jenkins. >> Where "get it working" would be to have the <system-out> nested under >> <testcase> so it can do the correct association. >> >> I have no idea who else supports this format though. It's hard to predict >> given that there is no schema for this XML file. >> >> -- >> Luke Daley >> Principal Engineer, Gradleware >> http://gradleware.com >> >> Join me at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: >> http://www.gradlesummit.com >> >> >> --------------------------------------------------------------------- >> To unsubscribe from this list, please visit: >> >> http://xircles.codehaus.org/manage_email >> >> > > > -- > Adam Murdoch > Gradle Co-founder > http://www.gradle.org > VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting > http://www.gradleware.com > > Join us at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: > http://www.gradlesummit.com > -- Luke Daley Principal Engineer, Gradleware http://gradleware.com Join me at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: http://www.gradlesummit.com --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
