This is now in 1.7. I've verified that it does what I needed it to with Jenkins, which was to have the https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Attachments+Plugin plugin associate Selenium screen shots with individual test methods instead of at the class level (which is useless).
On 24/04/2013, at 1:27 PM, Luke Daley <luke.da...@gradleware.com> wrote: > > On 23/04/2013, at 11:00 PM, Adam Murdoch <adam.murd...@gradleware.com> wrote: > >> >> The longer-term plan is to deliver the test events to the CI server via the >> tooling API, and we can make these events as rich as we like. Doesn't help >> us now, though. >> >> There are 3 case we want to support here: >> >> 1. Don't generate the XML at all. For example, don't generate the xml if >> nothing went wrong, or if I'm doing a dev build, or if I'm using a CI server >> that doesn't use it (TeamCity uses the events already, Bamboo doesn't show >> the output). >> 2. Generate the XML with output per atomic test. >> 3. Generate the XML with output per test class (aka 'broken mode'). >> >> We want to deprecate and remove #3 as soon as is practical, once #2 is >> available. The question here is what we would break if we just switched to >> #2? We should put some effort into answering that question. > > We may have to do #2 and #3 indefinitely depending on what the tools out > there do. > > We need to work out the list of things we need to test. Here's what I can > think of (rough priority order): > > 1. Jenkins > 2. Hudson > 3. Ant JUnit HTML reporter > 4. Team City > 5. Bamboo > 6. Thoughtworks Go > > > There are a lot of CI servers out there, we can't test them all. > > I'll try to get in touch with someone from the Jenkins team and ask them > about why they support this format and what produces XML files in this > format. If something that is in widespread use already does this, then that's > a good sign. > > >> >> On 19/04/2013, at 10:33 PM, Luke Daley <luke.da...@gradleware.com> wrote: >> >>> Hi all, >>> >>> In the “JUnit XML” file that we currently produce we do not associate >>> output (std, err) with individual test cases, but with the suite (in >>> practice: class) as a whole. This is causing problems in integrating with >>> https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Attachments+Plugin. >>> >>> Given: >>> >>> package junit; >>> >>> import org.junit.Test; >>> import org.junit.runner.RunWith; >>> import org.junit.runners.JUnit4; >>> >>> @RunWith(JUnit4.class) >>> public class TheTest { >>> >>> @Test >>> public void t() { >>> System.out.println("foo"); >>> } >>> >>> } >>> >>> We generate XML like: >>> >>> <?xml version="1.1" encoding="UTF-8"?> >>> <testsuite name="junit.TheTest" tests="1" failures="0" errors="0" >>> timestamp="1970-01-01T00:00:00" hostname="hasdrubal.local" >>> time="1.366372749405E9"> >>> <properties/> >>> <testcase name="t" classname="junit.TheTest" time="0.001"/> >>> <system-out><![CDATA[foo >>> ]]></system-out> >>> <system-err><![CDATA[]]></system-err> >>> </testsuite> >>> >>> After digging through the Jenkins code for how it parses these files, I >>> discovered that it supports: >>> >>> <?xml version="1.1" encoding="UTF-8"?> >>> <testsuite name="junit.TheTest" tests="1" failures="0" errors="0" >>> timestamp="1970-01-01T00:00:00" hostname="hasdrubal.local" >>> time="1.366372749405E9"> >>> <properties/> >>> <testcase name="t" classname="junit.TheTest" time="0.001"> >>> <system-out><![CDATA[foo >>> ]]></system-out> >>> <system-err><![CDATA[]]></system-err> >>> </testcase> >>> </testsuite> >>> >>> That is, you can associate output with an individual testcase. >>> >>> I did some experiments with Maven to see what it does. Maven doesn't record >>> any stdio information in the XML file usually; it puts text files on the >>> side. However, I happened upon a case where it does. If you try and execute >>> a non public class you get: >>> >>> <?xml version="1.0" encoding="UTF-8" ?> >>> <testsuite failures="0" time="0.006" errors="2" skipped="0" tests="2" >>> name="junit.TheTest"> >>> >>> <testcase time="0.002" classname="junit.TheTest" name="initializationError"> >>> <error message="Test class should have exactly one public constructor" >>> type="java.lang.Exception">java.lang.Exception: Test class should have >>> exactly one public constructor >>> at >>> org.junit.runners.BlockJUnit4ClassRunner.validateOnlyOneConstructor(BlockJUnit4ClassRunner.java:131) >>> … >>> </error> >>> <system-out>@SLTests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time >>> elapsed: 0.033 sec >>> @SLRunning junit.TheTest >>> </system-out> >>> </testcase> >>> <testcase time="0.001" classname="junit.TheTest" name="initializationError"> >>> <error message="Class junit.TheTest should be public" >>> type="java.lang.Exception">java.lang.Exception: Class junit.TheTest should >>> be public >>> at >>> org.junit.runners.model.FrameworkMethod.validatePublicVoid(FrameworkMethod.java:87) >>> … >>> </error> >>> </testcase> >>> </testsuite> >>> >>> The <system-out> elements are nested under <testcase> there. So it seems >>> there is some precedent for this. >>> >>> >>> I have some incentive to at least get this working for a client in Jenkins. >>> Where "get it working" would be to have the <system-out> nested under >>> <testcase> so it can do the correct association. >>> >>> I have no idea who else supports this format though. It's hard to predict >>> given that there is no schema for this XML file. >>> >>> -- >>> Luke Daley >>> Principal Engineer, Gradleware >>> http://gradleware.com >>> >>> Join me at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: >>> http://www.gradlesummit.com >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe from this list, please visit: >>> >>> http://xircles.codehaus.org/manage_email >>> >>> >> >> >> -- >> Adam Murdoch >> Gradle Co-founder >> http://www.gradle.org >> VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting >> http://www.gradleware.com >> >> Join us at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: >> http://www.gradlesummit.com >> > > -- > Luke Daley > Principal Engineer, Gradleware > http://gradleware.com > > Join me at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: > http://www.gradlesummit.com > -- Luke Daley Principal Engineer, Gradleware http://gradleware.com Join me at the Gradle Summit 2013, June 13th and 14th in Santa Clara, CA: http://www.gradlesummit.com --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
