Re: Improving CI ITs
Jason van Zyl wrote: Benjamin, you've basically done all the work for the ITs so far. You think you might sketch out in the wiki what you think would be ideal? At this point I think we defer to you as you've been working on them so long. There isn't really much to say. The ITs just work as before my changes, merely a little more stable and faster (hopefully), which doesn't mean there isn't more to improve. Still on my TODO list is a patch for VELO's nice emma4it-maven-plugin to spit the XML/text reports for the coverage out during a normal build such that we don't need to generate a site but can simply activate a profile that adds the Emma invocations around the normal test execution. The next step would be to combine the test coverage from the core ITs with the coverage from the ordinary unit tests to get the complete picture. So far I have no clear idea how we arrange that workflow given that the ITs and the UTs reside in independent builds/directories. One option might be to collect the coverage data in a known location like java.io.tmpdir between the two builds but that's just a spontenous thought. If anybody feels there should be more to say, please ask ;-) Benjamin - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Improving CI ITs
On Tue, Dec 16, 2008 at 1:51 AM, Jason van Zyl jvan...@sonatype.com wrote: Howdy, This is primarily directed at Benjamin (who has done a great job getting the ITs in shape), John (who has spent a great deal of time getting our grid up and running), and Tom (who has made some very cool additions to Hudson which we will leverage for Maven). Today we started being able to build Maven 3.x across the grid but now it's time to get serious about doing some layering for speed and completeness. Tom has some very cool additions to Hudson that allow us to write plugins in Hudson as Plexus components and one of the first plugins Tom has written is a way to propagate a successful bootstrap of Maven across a set of projects. This plugin combined with the greatly improved ITs for the core and the fleshing out of the ITs for all the core plugins will allow us to really start hammering down on 3.x. By layering I think that a canonical system should be picked, say Ubuntu, and that should be a fast compile-only smoke test to look for breakages. If that works trigger the tests, if that works propagate to the grid, do the build and run the ITs. Then layer in plugin tests maybe daily. That may not be the perfect workflow and we should put this in the wiki. Once we pick a work flow we can probably write any plugins we need. We can also start selectively taking any useful components from Continuum as they should now work in Hudson. The Plexus integration was checked into Hudson about a month ago by Tom. Tom/John, I don't know how long it will take to finish the maven integration testing plugin and get all the nodes updated. But I can do the dirty trick that I did before which was the let the bootstrap feed into a predefined Maven installation location. I just want to get the ITs triggered correctly using the version of Maven built by the bootstrap. Actually what was checked in to Hudson is a way to plug into the plugin mechanism itself. The Plexus integration is a possible implementation of that, but is not publicly available. The Hudson plugin for maven integration testing that I wrote is not based on the Plexus integration. It provides two things: - a build trigger that monitors for the availability of a new Maven version (e.g. a new snapshot or new RC) - a Maven builder that downloads and installs the Maven version prior to running a build Important to note is that this will work too for a Maven that is built on another (remote) Hudson. So anyone could configure it to track nightly builds or release candidates, and continously build their own projects using the latest Maven version. And of course it can be used too to build the maven ITs themselves. There's still a little work to do. - For starters it currently depends on having the Maven build as an archived artifact in Hudson, but perhaps it's more desirable to get them from a Maven repository ? - There should be a clear way to identify which Maven version was used - It's not threadsafe yet, so multiple concurrent builds with the same builder would definitely go wrong. Tom Benjamin, you've basically done all the work for the ITs so far. You think you might sketch out in the wiki what you think would be ideal? At this point I think we defer to you as you've been working on them so long. Thanks, Jason -- Jason van Zyl Founder, Apache Maven jason at sonatype dot com -- In short, man creates for himself a new religion of a rational and technical order to justify his work and to be justified in it. -- Jacques Ellul, The Technological Society - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Improving CI ITs
On 17-Dec-08, at 5:00 PM, Tom Huybrechts wrote: On Tue, Dec 16, 2008 at 1:51 AM, Jason van Zyl jvan...@sonatype.com wrote: Howdy, This is primarily directed at Benjamin (who has done a great job getting the ITs in shape), John (who has spent a great deal of time getting our grid up and running), and Tom (who has made some very cool additions to Hudson which we will leverage for Maven). Today we started being able to build Maven 3.x across the grid but now it's time to get serious about doing some layering for speed and completeness. Tom has some very cool additions to Hudson that allow us to write plugins in Hudson as Plexus components and one of the first plugins Tom has written is a way to propagate a successful bootstrap of Maven across a set of projects. This plugin combined with the greatly improved ITs for the core and the fleshing out of the ITs for all the core plugins will allow us to really start hammering down on 3.x. By layering I think that a canonical system should be picked, say Ubuntu, and that should be a fast compile-only smoke test to look for breakages. If that works trigger the tests, if that works propagate to the grid, do the build and run the ITs. Then layer in plugin tests maybe daily. That may not be the perfect workflow and we should put this in the wiki. Once we pick a work flow we can probably write any plugins we need. We can also start selectively taking any useful components from Continuum as they should now work in Hudson. The Plexus integration was checked into Hudson about a month ago by Tom. Tom/John, I don't know how long it will take to finish the maven integration testing plugin and get all the nodes updated. But I can do the dirty trick that I did before which was the let the bootstrap feed into a predefined Maven installation location. I just want to get the ITs triggered correctly using the version of Maven built by the bootstrap. Actually what was checked in to Hudson is a way to plug into the plugin mechanism itself. The Plexus integration is a possible implementation of that, but is not publicly available. We can make that publicly available. Nothing special in there. The Hudson plugin for maven integration testing that I wrote is not based on the Plexus integration. It provides two things: - a build trigger that monitors for the availability of a new Maven version (e.g. a new snapshot or new RC) - a Maven builder that downloads and installs the Maven version prior to running a build Important to note is that this will work too for a Maven that is built on another (remote) Hudson. So anyone could configure it to track nightly builds or release candidates, and continously build their own projects using the latest Maven version. And of course it can be used too to build the maven ITs themselves. There's still a little work to do. - For starters it currently depends on having the Maven build as an archived artifact in Hudson, but perhaps it's more desirable to get them from a Maven repository ? +1 It would nice to just say I would like Maven 2.0.9 and Hudson just grabs it and installs it. - There should be a clear way to identify which Maven version was used - It's not threadsafe yet, so multiple concurrent builds with the same builder would definitely go wrong. Tom Benjamin, you've basically done all the work for the ITs so far. You think you might sketch out in the wiki what you think would be ideal? At this point I think we defer to you as you've been working on them so long. Thanks, Jason -- Jason van Zyl Founder, Apache Maven jason at sonatype dot com -- In short, man creates for himself a new religion of a rational and technical order to justify his work and to be justified in it. -- Jacques Ellul, The Technological Society - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org Thanks, Jason -- Jason van Zyl Founder, Apache Maven jason at sonatype dot com -- - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Improving CI ITs
On 16/12/2008, at 11:51 AM, Jason van Zyl wrote: By layering I think that a canonical system should be picked, say Ubuntu, and that should be a fast compile-only smoke test to look for breakages. If that works trigger the tests, if that works propagate to the grid, do the build and run the ITs. Then layer in plugin tests maybe daily. That may not be the perfect workflow and we should put this in the wiki. +1, something like this sounds good. Tom/John, I don't know how long it will take to finish the maven integration testing plugin and get all the nodes updated. But I can do the dirty trick that I did before which was the let the bootstrap feed into a predefined Maven installation location. I just want to get the ITs triggered correctly using the version of Maven built by the bootstrap. I still don't quite get the approaches being used here. Can someone clear up all the pieces for me? - we have deployment of the IT plugins to a repository even as snapshots, though I thought it was required to be pre-installed based on the bits below? - maven-integration-tests checks out the ITs from trunk but how does that integrate with Hudson's existing SCM checks? - Hudson plugins are meant to coordinate running with a version instead of requiring the above? I agree there needs to be one standard way to do these things, but I'd like to understand the technical reasons for choosing the bootstrap, etc. over the unpack-and-run approach of maven-core-it-runner before it disappears completely. The only one I've heard described was the possibility of the IT artifacts in the repo being out of date, but that's irrelevant in a CI situation and they seem to be getting deployed above anyway. To me, the approach is much more Maven-y by reusing the repository and ordering, and it's easy to understand. Where did it not address the needs in setting this up? I also think it makes sense to release the integration tests that were used to verify a particular release of Maven so we always have that locked in. Otherwise as the ITs change, we have a different measure of what worked in 2.0.x and what didn't. The runner approach is much better suited to this rather than checking it out from a tag each time. Benjamin, you've basically done all the work for the ITs so far. You think you might sketch out in the wiki what you think would be ideal? At this point I think we defer to you as you've been working on them so long. Benjamin++ :) Cheers, Brett -- Brett Porter br...@apache.org http://blogs.exist.com/bporter/ - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
RE: Improving CI ITs
- we have deployment of the IT plugins to a repository even as snapshots, though I thought it was required to be pre-installed based on the bits below? Everything is ready now for them to be deployed as releases to a repo that is separate from Central to avoid the clashes that people saw before. but I'd like to understand the technical reasons for choosing the bootstrap, etc. over the unpack-and-run approach of maven-core-it-runner before it disappears completely. The only one I've heard described was the possibility of the IT artifacts in the repo being out of date, but that's irrelevant in a CI situation and they seem to be getting deployed above anyway. John and I came up with this solution because it was frustrating always trying to chase either the maven snapshot or the it snapshot when things were committed. By putting it into the bootstrap, we now have a single execution that builds both Maven and the Its and then run them together. A single atomic unit. I still see some value in the unpack-and-run approach personally but I also think having multiple ways to run Its is annoying to explain to new users. I think the bootstrap one is easy. I also think it makes sense to release the integration tests that were used to verify a particular release of Maven so we always have that locked in. Otherwise as the ITs change, we have a different measure of what worked in 2.0.x and what didn't. The runner approach is much better suited to this rather than checking it out from a tag each time. Agreed. - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: Improving CI ITs
On 17/12/2008, at 1:59 PM, Brian E. Fox wrote: - we have deployment of the IT plugins to a repository even as snapshots, though I thought it was required to be pre-installed based on the bits below? Everything is ready now for them to be deployed as releases to a repo that is separate from Central to avoid the clashes that people saw before. Great. I wonder, is the test still valuable enough if we kick off a Jetty instance and host the repo locally rather than relying on an external resource being up/accessible? Upsides: that would also help with avoiding repo manager interference in the results, tests would be faster Downsides: a bit more complicated test set up. If it were started from the POM, it disadvantages IDE running, if it were started from the tests it adds a bit of complication in there. WDYT? John and I came up with this solution because it was frustrating always trying to chase either the maven snapshot or the it snapshot when things were committed. By putting it into the bootstrap, we now have a single execution that builds both Maven and the Its and then run them together. A single atomic unit. I still see some value in the unpack-and-run approach personally but I also think having multiple ways to run Its is annoying to explain to new users. I think the bootstrap one is easy. It might be nice to investigate a CI plugin that can always keep those builds together and take another look in the future. There are still some things that make me a bit uneasy about it: - the dummy POM (2.0.x)... seems like this should be functionality right in the top level POM or a proper module. - it seems awkward to work on , you'd need to revert to running mvn from your working directory for the IT instead since you have to commit for the other changes to take, or reconfigure itDir - it's sad that the main way to build Maven is not with Maven :) I also think it makes sense to release the integration tests that were used to verify a particular release of Maven so we always have that locked in. Otherwise as the ITs change, we have a different measure of what worked in 2.0.x and what didn't. The runner approach is much better suited to this rather than checking it out from a tag each time. Agreed. Cool. I see this change still supports that through a tagged checkout. Will we release the ITs for 2.0.10, or is that too soon? Cheers, Brett -- Brett Porter br...@apache.org http://blogs.exist.com/bporter/ - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
