Re: Integration testing
cool stuff - thanks! On 18.09.2014, at 08:06, Manfred Moser wrote: > I recorded a quick video that shows the integration testing setup in the > Android Maven plugin using the Takari plugin testing and the Maven > development IDE tooling to help the committers on the plugin. > > Might be interesting for others as well.. > > https://www.youtube.com/watch?v=PkLUgUyx8Kc > > manfred > > - > To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org > For additional commands, e-mail: dev-h...@maven.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
RE: Integration Testing for WAGON-73
Curious, I only get 2 failures and one error (against 2.0.7 without the patch): Results : Failed tests: testit0045(org.apache.maven.integrationtests.MavenIT0045Test) testit0046(org.apache.maven.integrationtests.MavenIT0046Test) Tests in error: testit0104(org.apache.maven.integrationtests.MavenIT0104Test) Tests run: 115, Failures: 2, Errors: 1, Skipped: 0 And I get exactly the same errors and failures against 2.0.7 with the patch. I wonder why you are getting so many other failures Brian? What other enviromental settings should I have or not have? I have a mirror for central (Proximity Server that hosts Central, Codehaus and our internal repository). And a snapshot repository defined in our always on profile. William - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Integration Testing for WAGON-73
This test is definitely flakey. I was able to fix it in eclipse unit
testing but then it failed again on core.
Build the wagon and place it in maven2/lib. Then edit the m2.conf file:
"main is org.apache.maven.cli.MavenCli from plexus.core
set maven.home default ${user.home}/m2
[plexus.core]
load ${maven.home}/lib/wagon-manager-2.0-SNAPSHOT.jar
load ${maven.home}/lib/*.jar"
This loads the wagon before the one from the uberjar.
I'm out of time for tonight, I'm in EST (GMT -5) and it's almost
midnight so the tests will start passing again anyway...but more
importantly I'm tired ;-)
-Original Message-
From: William Ferguson [mailto:[EMAIL PROTECTED]
Sent: Monday, August 06, 2007 11:29 PM
To: Maven Developers List
Subject: Integration Testing for WAGON-73
I've moved this discussion off JIRA.
Brian,
I have just checked out the core-integration-testing/trunk and was just
trying to execute the core-integration-tests myself. How did you execute
them for Maven-2.0.7 with the WAGON-73 patch applied?
Did you need to change something in the core-integration-tests POM?
I looked for a document describing what needed to be done to configure
the integration-tests but didn't find anything. Maybe you could point me
at something.
William
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Integration Testing
my only recommendation would be to take a look at the maven-assembly-plugin (SVN: http://svn.apache.org/repos/asf/maven/plugins/trunk/maven-assembly-plugin) and go for something like that... On 4/16/07, Barrie Treloar <[EMAIL PROTECTED]> wrote: On 4/17/07, John Casey <[EMAIL PROTECTED]> wrote: > Jerome, that link contains information on the strategy I've been using in > the assembly plugin, and it seems to work well (although it's a bit slower, > since it's spawning so many Maven builds). I'm hoping to converge the tools > we're using in the core ITs with those in the maven-component-test-plugin > (used to be maven-plugin-test-plugin), and the maven-invoker-plugin, and > arrive at some single solution (a plugin) that we can use everywhere...but I > haven't even been able to come up with a unified feature list yet. So, does anyone have a recommended approach so I can move forward in the mean time? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Integration Testing
On 4/17/07, John Casey <[EMAIL PROTECTED]> wrote: Jerome, that link contains information on the strategy I've been using in the assembly plugin, and it seems to work well (although it's a bit slower, since it's spawning so many Maven builds). I'm hoping to converge the tools we're using in the core ITs with those in the maven-component-test-plugin (used to be maven-plugin-test-plugin), and the maven-invoker-plugin, and arrive at some single solution (a plugin) that we can use everywhere...but I haven't even been able to come up with a unified feature list yet. So, does anyone have a recommended approach so I can move forward in the mean time? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Integration Testing
Jerome, that link contains information on the strategy I've been using in the assembly plugin, and it seems to work well (although it's a bit slower, since it's spawning so many Maven builds). I'm hoping to converge the tools we're using in the core ITs with those in the maven-component-test-plugin (used to be maven-plugin-test-plugin), and the maven-invoker-plugin, and arrive at some single solution (a plugin) that we can use everywhere...but I haven't even been able to come up with a unified feature list yet. -john On 4/15/07, Jerome Lacoste <[EMAIL PROTECTED]> wrote: On 2/24/07, Brett Porter <[EMAIL PROTECTED]> wrote: > On 04/01/2007, at 4:32 PM, Brett Porter wrote: > > > Jason - any further thoughts on this? > > Ping... No is a valid answer :) > > I'd like to get your summary put somewhere individuals can pick > things off to work on - probably a jira project for shared. WDYT? > > I'm overcommitted for working on things right now, but I know a > couple of people are confused about the IT testing, and we've got all > those chronically broken plugins. Volunteers? I want to improve the state of IT for the mojos I work with (in particular in the mojo project). I am particularly interested in doing it in an sort of official/standard way. - [ ] We need to standardize on integration testing in general. We have people going all over the place and it's a disaster. - [ ] We have too many IT plugins (3) - [ ] We have too many invokers (5) - [ ] We have too many verifiers (3) I've found this: http://maven.apache.org/developers/committer-testing-plugins.html Does that represent the current IT strategy 'standard' ? Was that written after the point above was resolved ? Cheers, Jerome - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Integration Testing
On 2/24/07, Brett Porter <[EMAIL PROTECTED]> wrote: On 04/01/2007, at 4:32 PM, Brett Porter wrote: > Jason - any further thoughts on this? Ping... No is a valid answer :) I'd like to get your summary put somewhere individuals can pick things off to work on - probably a jira project for shared. WDYT? I'm overcommitted for working on things right now, but I know a couple of people are confused about the IT testing, and we've got all those chronically broken plugins. Volunteers? I want to improve the state of IT for the mojos I work with (in particular in the mojo project). I am particularly interested in doing it in an sort of official/standard way. - [ ] We need to standardize on integration testing in general. We have people going all over the place and it's a disaster. - [ ] We have too many IT plugins (3) - [ ] We have too many invokers (5) - [ ] We have too many verifiers (3) I've found this: http://maven.apache.org/developers/committer-testing-plugins.html Does that represent the current IT strategy 'standard' ? Was that written after the point above was resolved ? Cheers, Jerome - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Integration Testing
On 2/24/07, Jason van Zyl <[EMAIL PROTECTED]> wrote: > I'd like to get your summary put somewhere individuals can pick > things off to work on - probably a jira project for shared. WDYT? > I put the summary here for now, it's pretty elaborate and can most likely be worked on easily. Given one or two people have expressed interested or worked on the ITs I think that will suffice. http://svn.apache.org/repos/asf/maven/core-integration-testing/trunk/ ITProblems.txt Is there something that describes how to setup IT tests for plugins and how to go about it? I'm trying to use TDD to verify that the plugin I am testing, is correctly attaching the write output to the project. via setArtifact and addAttachedArtifact. I'm not sure how to go about doing this. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Integration Testing
On 23 Feb 07, at 10:08 PM 23 Feb 07, Brett Porter wrote: On 04/01/2007, at 4:32 PM, Brett Porter wrote: Jason - any further thoughts on this? Ping... No is a valid answer :) I'd like to get your summary put somewhere individuals can pick things off to work on - probably a jira project for shared. WDYT? I put the summary here for now, it's pretty elaborate and can most likely be worked on easily. Given one or two people have expressed interested or worked on the ITs I think that will suffice. http://svn.apache.org/repos/asf/maven/core-integration-testing/trunk/ ITProblems.txt The top things that could be done: 1) The issues that would be most helpful that could be tackled on a piecewise basis by many would be to take plugin specific ITs out of the ITs. There are many in there for the surefire plugin so while you're doing that you can look at it. Piecewise but probably totally simple because you have to replace it with an IT that actually tests what it was testing. A lot of time I have had to make a new IT plugin flavour. 2) The next issue of importance would be to collect all the in IT plugin plugins, invokers and verifiers and align all theses. 3) Once 2) is done then we wire the embedder option into the resulting invoker. As far as them working in situ: the ITs could now run from the top level of http://svn.apache.org/repos/asf/maven/core-integration- testing/trunk/ with an addition to the POM. It's not there because I generally build from the top-level and then walk into http:// svn.apache.org/repos/asf/maven/core-integration-testing/trunk/core- integration-tests/. The invoker plugin has been released and that's what I used to install the supporting artifacts because they can't install in a reactor run normally because there are 5 artifacts with the id. I am not entirely happy with the structure we have but I don't think it's worth changing right now until 1) and 2) are done. It's not obvious how to run them from and IDE and that's where I've found it to be most convenient to run them. I'm overcommitted for working on things right now, but I know a couple of people are confused about the IT testing, and we've got all those chronically broken plugins. Volunteers? On 18/12/2006, at 4:35 PM, Brett Porter wrote: On 12/12/2006, at 5:08 AM, Jason van Zyl wrote: So, in response to John's email: I think we need to settle on what we're going to use and stop writing new tools. We have several invokers, several verifiers, several IT plugins, and the plugin harness. It is simply out of control. We have different methods being used in different plugins, nothing is standard and it's going to kill us. The most prevalent tool, as defective as it might be is what is being used in the ITs themselves. Stephane managed to use this successfully in some of his plugins. Then after that we have an array of usages. What should happen before we start writing more stuff is to figure out something we can use now, and how to merge what we have together instead of writing more tools. Agreed. I think that's what John was getting at too, but by doing it clean rather than rewriting something that was in use somewhere. So the work left to do, in either case, is apply it consistently and get rid of the stuff we aren't going to use. To me, it looks like: - the plugin-testing-harness needs to go. They should be integration tests that use a proper pom, or use pure mocks rather than the stubs that tend to just have a bunch of impossible-to- get-under-real-condition values. - John's test tools have the most complete invocation options, and tools for managing repositories that we can reuse, so I'd opt for that in that area - the verifier is well utilised, so if that is merged with the code from the verifier plugin then we can lose the invocation stuff and repository management stuff and merge it all together Can we have a wiki page with this work list? Or, can we check in the omni outliner files + an export for the non-mac users to review? [snip points I agree with] - [+] The ITs should be in a project of their own so that we can reuse them across versions of Maven. We could actually run new versions of integration tests against old versions of Maven. Solution: the ITs are now in a separate build and it is possible to run them How should this play out in plugins? I would be in favour of separate projects for these. Two things that are must haves for me: - integration tests / anything that forks a Maven instance must *not* be part of the normal test build for a plugin. They take way too long, and lead to use of maven.test.skip :( - as far as integration testing in general, I think we recommend a separate project, but enable them to be part of the same project for simplicity of development (this is more specific to things like web applications
Re: Integration Testing
On 04/01/2007, at 4:32 PM, Brett Porter wrote: Jason - any further thoughts on this? Ping... No is a valid answer :) I'd like to get your summary put somewhere individuals can pick things off to work on - probably a jira project for shared. WDYT? I'm overcommitted for working on things right now, but I know a couple of people are confused about the IT testing, and we've got all those chronically broken plugins. Volunteers? On 18/12/2006, at 4:35 PM, Brett Porter wrote: On 12/12/2006, at 5:08 AM, Jason van Zyl wrote: So, in response to John's email: I think we need to settle on what we're going to use and stop writing new tools. We have several invokers, several verifiers, several IT plugins, and the plugin harness. It is simply out of control. We have different methods being used in different plugins, nothing is standard and it's going to kill us. The most prevalent tool, as defective as it might be is what is being used in the ITs themselves. Stephane managed to use this successfully in some of his plugins. Then after that we have an array of usages. What should happen before we start writing more stuff is to figure out something we can use now, and how to merge what we have together instead of writing more tools. Agreed. I think that's what John was getting at too, but by doing it clean rather than rewriting something that was in use somewhere. So the work left to do, in either case, is apply it consistently and get rid of the stuff we aren't going to use. To me, it looks like: - the plugin-testing-harness needs to go. They should be integration tests that use a proper pom, or use pure mocks rather than the stubs that tend to just have a bunch of impossible-to-get- under-real-condition values. - John's test tools have the most complete invocation options, and tools for managing repositories that we can reuse, so I'd opt for that in that area - the verifier is well utilised, so if that is merged with the code from the verifier plugin then we can lose the invocation stuff and repository management stuff and merge it all together Can we have a wiki page with this work list? Or, can we check in the omni outliner files + an export for the non-mac users to review? [snip points I agree with] - [+] The ITs should be in a project of their own so that we can reuse them across versions of Maven. We could actually run new versions of integration tests against old versions of Maven. Solution: the ITs are now in a separate build and it is possible to run them How should this play out in plugins? I would be in favour of separate projects for these. Two things that are must haves for me: - integration tests / anything that forks a Maven instance must *not* be part of the normal test build for a plugin. They take way too long, and lead to use of maven.test.skip :( - as far as integration testing in general, I think we recommend a separate project, but enable them to be part of the same project for simplicity of development (this is more specific to things like web applications where it is probably beneficial). - [ ] We should be able to easily integrate the IT into a larger run where we can use forked or embedded execution. Not sure what you mean here. What is an IT that *doesn't* use forked or embedded execution? - [ ] automate the testing of ITs submitted by users what does this mean? I think, like a patch, a submitted IT still needs to be reviewed, and incorporated into the main test suite (with corresponding fix-for version so it only runs when it is expected to work). Otherwise, long term, we'll have lots of duplicated or poorly conceived ITs. I've seen test cases submitted that are quite useful at demonstrating something, but contain half of the user's proect which is not good for our long term scalability. - [ ] Each IT should have its own repository if it needs resources from repository. We can't mess with a users repository when testing. - [ ] We need to have a file system based remote repository for testing Agreed. Isn't that what John's tool already does? - [ ] We need to standardize on integration testing in general. We have people going all over the place and it's a disaster. - [ ] We have too many IT plugins (3) - [ ] We have too many invokers (5) - [ ] We have too many verifiers (3) Let's specifically get these mapped out and a path forward so that everyone can push towards it, rather than relying on you to do the work. - [ ] The ITs should run nicely from an IDE. Solution: this does work but requires that you run mvn clean resources:testResources first as the IDE doesn't know how to set that up. Needs to be fully fixed. But it is much nicer running this stuff in your IDE. Agree with th
Re: Integration Testing
Jason - any further thoughts on this? On 18/12/2006, at 4:35 PM, Brett Porter wrote: On 12/12/2006, at 5:08 AM, Jason van Zyl wrote: So, in response to John's email: I think we need to settle on what we're going to use and stop writing new tools. We have several invokers, several verifiers, several IT plugins, and the plugin harness. It is simply out of control. We have different methods being used in different plugins, nothing is standard and it's going to kill us. The most prevalent tool, as defective as it might be is what is being used in the ITs themselves. Stephane managed to use this successfully in some of his plugins. Then after that we have an array of usages. What should happen before we start writing more stuff is to figure out something we can use now, and how to merge what we have together instead of writing more tools. Agreed. I think that's what John was getting at too, but by doing it clean rather than rewriting something that was in use somewhere. So the work left to do, in either case, is apply it consistently and get rid of the stuff we aren't going to use. To me, it looks like: - the plugin-testing-harness needs to go. They should be integration tests that use a proper pom, or use pure mocks rather than the stubs that tend to just have a bunch of impossible-to-get- under-real-condition values. - John's test tools have the most complete invocation options, and tools for managing repositories that we can reuse, so I'd opt for that in that area - the verifier is well utilised, so if that is merged with the code from the verifier plugin then we can lose the invocation stuff and repository management stuff and merge it all together Can we have a wiki page with this work list? Or, can we check in the omni outliner files + an export for the non-mac users to review? [snip points I agree with] - [+] The ITs should be in a project of their own so that we can reuse them across versions of Maven. We could actually run new versions of integration tests against old versions of Maven. Solution: the ITs are now in a separate build and it is possible to run them How should this play out in plugins? I would be in favour of separate projects for these. Two things that are must haves for me: - integration tests / anything that forks a Maven instance must *not* be part of the normal test build for a plugin. They take way too long, and lead to use of maven.test.skip :( - as far as integration testing in general, I think we recommend a separate project, but enable them to be part of the same project for simplicity of development (this is more specific to things like web applications where it is probably beneficial). - [ ] We should be able to easily integrate the IT into a larger run where we can use forked or embedded execution. Not sure what you mean here. What is an IT that *doesn't* use forked or embedded execution? - [ ] automate the testing of ITs submitted by users what does this mean? I think, like a patch, a submitted IT still needs to be reviewed, and incorporated into the main test suite (with corresponding fix-for version so it only runs when it is expected to work). Otherwise, long term, we'll have lots of duplicated or poorly conceived ITs. I've seen test cases submitted that are quite useful at demonstrating something, but contain half of the user's proect which is not good for our long term scalability. - [ ] Each IT should have its own repository if it needs resources from repository. We can't mess with a users repository when testing. - [ ] We need to have a file system based remote repository for testing Agreed. Isn't that what John's tool already does? - [ ] We need to standardize on integration testing in general. We have people going all over the place and it's a disaster. - [ ] We have too many IT plugins (3) - [ ] We have too many invokers (5) - [ ] We have too many verifiers (3) Let's specifically get these mapped out and a path forward so that everyone can push towards it, rather than relying on you to do the work. - [ ] The ITs should run nicely from an IDE. Solution: this does work but requires that you run mvn clean resources:testResources first as the IDE doesn't know how to set that up. Needs to be fully fixed. But it is much nicer running this stuff in your IDE. Agree with the point, but not sure what you are referring to about testResources - the generated projects for IDEA and I think Eclipse already do this (and obviously better integration will bring it). I think the dangerous thing is using resources for non-classpath resources. It's better for the tests to setup and use a clean project instance for an IT itself (using helper tools). [snip specific notes on ol
Re: Integration Testing
On 18 Dec 06, at 9:18 AM 18 Dec 06, David Whitehurst wrote: Jason: Here! here! I recently joined the AppFuse bunch and being more the builder, packager, deployer-type, I'm working on AppFuse2 and the use of the archetype. Since, I've been on a large state project for Massachusetts, we have not used Maven, so my knowledge of Maven has been some small use of Maven1. I'm am so disappointed in my inability to just further or push the work on the AppFuse archetype. It's bad for me personally because I'm a senior level type but I can't just make good progress because I'm always having some trouble determining *what* I'm supposed to do to get things to work. AppFuse is difficult but Maven's even more difficult because I don't know where the knowledge home is for every plugin, mojo, etc. I don't know *how* I'm supposed to do things. It seems that Maven and the Plugin World needs a roadmap and some throttling of what gets created and what becomes public domain. Thanks Jason for your work behind a great development tool and your work in trying to slow things down. I try my best. http://www.amazon.com/Praise-Slowness-Worldwide-Movement-Challenging/ dp/006054578X Jason. David On 12/11/06, Jason van Zyl <[EMAIL PROTECTED]> wrote: Hi, This is in response to John's email about integration testing, some notes that I've taken while trying to clean up the integration testing, and some suggestions about where we go from here with respect to integration testing. You will see from the list below that there are many problems with the ITs ranging from integration tests for specific plugins being in there to using production dependencies, to having tons of duplicated for doing ITs. So, in response to John's email: I think we need to settle on what we're going to use and stop writing new tools. We have several invokers, several verifiers, several IT plugins, and the plugin harness. It is simply out of control. We have different methods being used in different plugins, nothing is standard and it's going to kill us. The most prevalent tool, as defective as it might be is what is being used in the ITs themselves. Stephane managed to use this successfully in some of his plugins. Then after that we have an array of usages. What should happen before we start writing more stuff is to figure out something we can use now, and how to merge what we have together instead of writing more tools. This is just to get the discussion started. I will take a first pass at enumerating the collection of tools we have and where they are used. But here are my notes from the main core ITs. Thus far as some of the ideas pertain to ITs in general like not polluting the local repo when testing .. Jason. --- - [-] Goals - [ ] An IT should be completely self-contained so that the problem can be understood by looking in one place, in one Maven project. - [ ] We should be able to create an Archetype so that users can easily create ITs for us - [+] The ITs should be in a project of their own so that we can reuse them across versions of Maven. We could actually run new versions of integration tests against old versions of Maven. Solution: the ITs are now in a separate build and it is possible to run them - [ ] We should be able to easily integrate the IT into a larger run where we can use forked or embedded execution. - [ ] We should create Archetypes for all categories of problems so that anyone can generate tests cases for us. Then there is so much that we can do in terms of automating this process of checking tests for quality along with the patches. - [ ] automate the testing of ITs submitted by users - [ ] Each IT should have its own repository if it needs resources from repository. We can't mess with a users repository when testing. - [ ] We need to have a file system based remote repository for testing - [ ] We need to standardize on integration testing in general. We have people going all over the place and it's a disaster. - [ ] We have too many IT plugins (3) - [ ] We have too many invokers (5) - [ ] We have too many verifiers (3) - [ ] The ITs should run nicely from an IDE. Solution: this does work but requires that you run mvn clean resources:testResources first as the IDE doesn't know how to set that up. Needs to be fully fixed. But it is much nicer running this stuff in your IDE. - [-] Problems with ITs - [+] Verifier jar required by the bootstrap requires a special verifier.jar there is no released version of this tool. Solution: the bootstrap now uses Ant and we've gotten rid of a lot of the complexity. - [+] The maven-core-it plugin needs to be d
Re: Integration Testing
Jason: Here! here! I recently joined the AppFuse bunch and being more the builder, packager, deployer-type, I'm working on AppFuse2 and the use of the archetype. Since, I've been on a large state project for Massachusetts, we have not used Maven, so my knowledge of Maven has been some small use of Maven1. I'm am so disappointed in my inability to just further or push the work on the AppFuse archetype. It's bad for me personally because I'm a senior level type but I can't just make good progress because I'm always having some trouble determining *what* I'm supposed to do to get things to work. AppFuse is difficult but Maven's even more difficult because I don't know where the knowledge home is for every plugin, mojo, etc. I don't know *how* I'm supposed to do things. It seems that Maven and the Plugin World needs a roadmap and some throttling of what gets created and what becomes public domain. Thanks Jason for your work behind a great development tool and your work in trying to slow things down. David On 12/11/06, Jason van Zyl <[EMAIL PROTECTED]> wrote: Hi, This is in response to John's email about integration testing, some notes that I've taken while trying to clean up the integration testing, and some suggestions about where we go from here with respect to integration testing. You will see from the list below that there are many problems with the ITs ranging from integration tests for specific plugins being in there to using production dependencies, to having tons of duplicated for doing ITs. So, in response to John's email: I think we need to settle on what we're going to use and stop writing new tools. We have several invokers, several verifiers, several IT plugins, and the plugin harness. It is simply out of control. We have different methods being used in different plugins, nothing is standard and it's going to kill us. The most prevalent tool, as defective as it might be is what is being used in the ITs themselves. Stephane managed to use this successfully in some of his plugins. Then after that we have an array of usages. What should happen before we start writing more stuff is to figure out something we can use now, and how to merge what we have together instead of writing more tools. This is just to get the discussion started. I will take a first pass at enumerating the collection of tools we have and where they are used. But here are my notes from the main core ITs. Thus far as some of the ideas pertain to ITs in general like not polluting the local repo when testing .. Jason. --- - [-] Goals - [ ] An IT should be completely self-contained so that the problem can be understood by looking in one place, in one Maven project. - [ ] We should be able to create an Archetype so that users can easily create ITs for us - [+] The ITs should be in a project of their own so that we can reuse them across versions of Maven. We could actually run new versions of integration tests against old versions of Maven. Solution: the ITs are now in a separate build and it is possible to run them - [ ] We should be able to easily integrate the IT into a larger run where we can use forked or embedded execution. - [ ] We should create Archetypes for all categories of problems so that anyone can generate tests cases for us. Then there is so much that we can do in terms of automating this process of checking tests for quality along with the patches. - [ ] automate the testing of ITs submitted by users - [ ] Each IT should have its own repository if it needs resources from repository. We can't mess with a users repository when testing. - [ ] We need to have a file system based remote repository for testing - [ ] We need to standardize on integration testing in general. We have people going all over the place and it's a disaster. - [ ] We have too many IT plugins (3) - [ ] We have too many invokers (5) - [ ] We have too many verifiers (3) - [ ] The ITs should run nicely from an IDE. Solution: this does work but requires that you run mvn clean resources:testResources first as the IDE doesn't know how to set that up. Needs to be fully fixed. But it is much nicer running this stuff in your IDE. - [-] Problems with ITs - [+] Verifier jar required by the bootstrap requires a special verifier.jar there is no released version of this tool. Solution: the bootstrap now uses Ant and we've gotten rid of a lot of the complexity. - [+] The maven-core-it plugin needs to be decoupled into its separate purposes because there are currently 12 different things going on in the plugin and it would be really confusing for a new user to figure out what's going on in the
Re: Integration Testing
On 12/12/2006, at 5:08 AM, Jason van Zyl wrote: So, in response to John's email: I think we need to settle on what we're going to use and stop writing new tools. We have several invokers, several verifiers, several IT plugins, and the plugin harness. It is simply out of control. We have different methods being used in different plugins, nothing is standard and it's going to kill us. The most prevalent tool, as defective as it might be is what is being used in the ITs themselves. Stephane managed to use this successfully in some of his plugins. Then after that we have an array of usages. What should happen before we start writing more stuff is to figure out something we can use now, and how to merge what we have together instead of writing more tools. Agreed. I think that's what John was getting at too, but by doing it clean rather than rewriting something that was in use somewhere. So the work left to do, in either case, is apply it consistently and get rid of the stuff we aren't going to use. To me, it looks like: - the plugin-testing-harness needs to go. They should be integration tests that use a proper pom, or use pure mocks rather than the stubs that tend to just have a bunch of impossible-to-get-under-real- condition values. - John's test tools have the most complete invocation options, and tools for managing repositories that we can reuse, so I'd opt for that in that area - the verifier is well utilised, so if that is merged with the code from the verifier plugin then we can lose the invocation stuff and repository management stuff and merge it all together Can we have a wiki page with this work list? Or, can we check in the omni outliner files + an export for the non-mac users to review? [snip points I agree with] - [+] The ITs should be in a project of their own so that we can reuse them across versions of Maven. We could actually run new versions of integration tests against old versions of Maven. Solution: the ITs are now in a separate build and it is possible to run them How should this play out in plugins? I would be in favour of separate projects for these. Two things that are must haves for me: - integration tests / anything that forks a Maven instance must *not* be part of the normal test build for a plugin. They take way too long, and lead to use of maven.test.skip :( - as far as integration testing in general, I think we recommend a separate project, but enable them to be part of the same project for simplicity of development (this is more specific to things like web applications where it is probably beneficial). - [ ] We should be able to easily integrate the IT into a larger run where we can use forked or embedded execution. Not sure what you mean here. What is an IT that *doesn't* use forked or embedded execution? - [ ] automate the testing of ITs submitted by users what does this mean? I think, like a patch, a submitted IT still needs to be reviewed, and incorporated into the main test suite (with corresponding fix-for version so it only runs when it is expected to work). Otherwise, long term, we'll have lots of duplicated or poorly conceived ITs. I've seen test cases submitted that are quite useful at demonstrating something, but contain half of the user's proect which is not good for our long term scalability. - [ ] Each IT should have its own repository if it needs resources from repository. We can't mess with a users repository when testing. - [ ] We need to have a file system based remote repository for testing Agreed. Isn't that what John's tool already does? - [ ] We need to standardize on integration testing in general. We have people going all over the place and it's a disaster. - [ ] We have too many IT plugins (3) - [ ] We have too many invokers (5) - [ ] We have too many verifiers (3) Let's specifically get these mapped out and a path forward so that everyone can push towards it, rather than relying on you to do the work. - [ ] The ITs should run nicely from an IDE. Solution: this does work but requires that you run mvn clean resources:testResources first as the IDE doesn't know how to set that up. Needs to be fully fixed. But it is much nicer running this stuff in your IDE. Agree with the point, but not sure what you are referring to about testResources - the generated projects for IDEA and I think Eclipse already do this (and obviously better integration will bring it). I think the dangerous thing is using resources for non-classpath resources. It's better for the tests to setup and use a clean project instance for an IT itself (using helper tools). [snip specific notes on old ITs that need to be updated] - [ ] artifactIds should be aligned with directories Agreed.
RE: integration testing thought
They absolutely do run through Maven itself. In the new integration testing scheme, the integration tests become simple JUnit tests, which, IMO, makes them even easier to run than Mojos... you can just press Play in Eclipse/IDEA and get a nice red/green report. The most important advantage to this is that since they're just JUnit tests, you have the full power of Java to use in configuring, running, and verifying your test output. All those 1000-2000 range tests that were "too complicated" before will just become JUnit tests run nightly. -Dan -Original Message- From: Brett Porter [mailto:[EMAIL PROTECTED] Sent: Thursday, October 19, 2006 4:58 AM To: Maven Developers List Subject: integration testing thought Hi, I haven't had a chance to look at the new bootstrap/integration testing stuff (it seems like the build is broken according to CI?), but had a thought I wanted to raise. Will the integration tests be runnable through Maven itself? ie, can the verifier be wrapped in a mojo (which I figured would be necessary to merge with the verifier plugin as Jason was discussing). The reason I suggest this is that it would make running all or individual integration tests simpler (cd, m2 integration-test; or use the reactor). It would also allow us to do code coverage for integration tests and so on. Thoughts? - Brett - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] ___ Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that may be confidential, proprietary, copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: integration testing thought
On 19 Oct 06, at 7:58 AM 19 Oct 06, Brett Porter wrote: Hi, I haven't had a chance to look at the new bootstrap/integration testing stuff (it seems like the build is broken according to CI?), but had a thought I wanted to raise. Will the integration tests be runnable through Maven itself? ie, can the verifier be wrapped in a mojo (which I figured would be necessary to merge with the verifier plugin as Jason was discussing). The reason I suggest this is that it would make running all or individual integration tests simpler (cd, m2 integration-test; or use the reactor). It would also allow us to do code coverage for integration tests and so on. Thoughts? This will work with the new setup. Each rest can be run individually and as part of one test suite so these sorts of things will be entirely possible. Dan will hopefully be aboard soon and he's created a very nice setup with is very testing/automation friendly. Once he's a committer I'll let him document it :-) Jason. - Brett - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Integration testing plugins
> -Original Message- > From: Kenney Westerhof [mailto:[EMAIL PROTECTED] > Sent: vendredi 11 août 2006 13:05 > To: Maven Developers List > Subject: Re: Integration testing plugins > > > > Vincent Massol wrote: > > > Also, Kenney do you know if the it plugin can now be used to test a > plugin > > that is being built for the first time? I remember that I didn't > activate it > > on the clover plugin because it would work only if the clover plugin was > > installed in the local repository and this came after integration > testing. > > Thus you would always run it tests on the previous version of the plugin > > (looked up from the local repo). I think I had a jira issue for this. > I'm > > offline now but I'll try to dig it up if need be. > > The following issue [1] should provide that: it was blocking the use of > the it plugin; once fixed, it worked fine. If the local repository > version of the plugin is used instead, then I suggest we re-open the > issue. > > [1] http://jira.codehaus.org/browse/MNG-870 Thanks Kenney for your answer. I've tried it and I have 3 problems: 1) It seems the it plugin is not available on ibiblio and thus the build failed when I added it to the clover plugin: org.apache.maven.plugins maven-it-plugin clean,install,clean,site integration-test test 2) My directory structure is as follows in the clover plugin: src/it/ |_ multiproject/ |_ jar1/ |_ ear1/ |_ [...] |_ pom.xml |_ simple/ |_ [...] |_ pom.xml When I ran the test goal of the it plugin it only executed the top level multiproject/pom.xm but not the modules defined in it. 3) When I removed the clover plugin from my local repo and then ran "mvn clean install" on the clover plugin it looked up the clover plugin from the remote repo (ibiblio) for running the tests. This is not the right behavior as the it plugin would need to use the clover plugin just built rather than the latest version available in the remot repo. Should I open JIRA issues for these? Thanks -Vincent p5.vert.ukl.yahoo.com uncompressed Wed Aug 23 12:27:00 GMT 2006 ___ Découvrez un nouveau moyen de poser toutes vos questions quelque soit le sujet ! Yahoo! Questions/Réponses pour partager vos connaissances, vos opinions et vos expériences. http://fr.answers.yahoo.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Integration testing plugins
Actually, pointing at a different local repo for the purposes of a build is trivial. However, in the case of testing a plugin, you have to install the plugin somewhere, then run a build that uses that plugin...this is not the case with anything other than Maven components, which makes it an exceptional use case. If Geronimo wants to fire off a build during their integration tests, they will be able to specify the local repository location without a problem, using either the embedder or the invoker. Once we get nearer a release of the new embedder, I'd like to converge the two, and hopefully get it down to a fork flag to determine which is used. But the plugin-test stuff is different, and it's an exotic use case. Also, if we're going to stop touching the local repository during a build, when will we use it? If you were to use the repository-assembly stuff from the assembly plugin to generate a one-time use repository containing all of your dependencies, then I can imagine forking off a local repository for integration testing. However, unless you're on a busy build server where you could reasonably expect to encounter race conditions on artifact resolution by way of the local repo, I would really hope that Maven would resolve to the same set of artifacts over two successive builds like this. I do think this would be a good option for many people when running integration tests (for instance, on large Continuum deployments). But for many users, this will be an unnecessary performance degradation. -john On 8/11/06, Jason van Zyl <[EMAIL PROTECTED]> wrote: On 11 Aug 06, at 11:25 AM 11 Aug 06, John Casey wrote: > it doesn't backup the whole local repository, only the files that > will be > affected by a plugin install. It should be relatively stable in > terms of the > backup/restore data transfer size...varying only by the number of > repositories that were consulted for that artifact (because there > is a set > of metadata files for each remote repository). > > I'm not sure what you mean by encapsulating things needed by the > runtime. > What I want to avoid is introducing excessive design changes to the > core > just so we can enable plugin manipulation that wouldn't happen in a > normal > use case. > I think this is a normal use case. A concrete example being the embedder pointing at a completely separate repository structure for, say, Geronimo. They use a Maven repository for their own use and you should be able to flip a parameter and use the encapsulated requirements for runtime: a repository, any configuration, any directories which hold configuration, caches, or anything of that nature. I think touching the repository that is actually used in the day-to-day is sort of fugly. If the plugin interacts with any artifacts then all those are going to be affected as so if you're doing integration tests on the Jetty plugin then you're going to pull in a whole raft of other dependencies, you're sure you're picking all that up and restoring it fully? I think some things with artifact resolution are a little loosy goosy at moment and not touching the users repository I think would be a better option. > -john > > On 8/11/06, Jason van Zyl <[EMAIL PROTECTED]> wrote: >> >> >> On 11 Aug 06, at 9:18 AM 11 Aug 06, John Casey wrote: >> >> > This should be solved by the stage/unstage mojos in the plugin-test >> > plugin. >> > That's its only current purpose. It will stage a new plugin >> > artifact into >> > the local repository, backing up what was there before, then >> clear the >> > pluginManager's cache of that plugin, in case it's been resolved >> > already. >> > Unstaging (de-staging?) will restore the backed up data, and clear >> > the cache >> > again. >> > >> >> We should move toward encapsulating anything the runtime requires so >> we could parameterize it on execution so that you don't have to >> tamper with the local repository. If someone has a massive local >> repository that could get pretty time consuming. >> >> > -john >> > >> > On 8/11/06, Kenney Westerhof <[EMAIL PROTECTED]> wrote: >> >> >> >> >> >> >> >> Vincent Massol wrote: >> >> >> >> > Also, Kenney do you know if the it plugin can now be used to >> test a >> >> plugin >> >> > that is being built for the first time? I remember that I didn't >> >> activate it >> >> > on the clover plugin because it would work only if the clover >> >> plugin was >> >> > installed in the local repository and this came after >> integration >> >> testing. >> >> > Thus you would always run it tests on the previous version of >> >> the plugin >> >> > (looked up from the local repo). I think I had a jira issue for >> >> this. >> >> I'm >> >> > offline now but I'll try to dig it up if need be. >> >> >> >> The following issue [1] should provide that: it was blocking the >> >> use of >> >> the it plugin; once fixed, it worked fine. If the local repository >> >> version of the plugin is used instead, then I suggest we re- >> open the >> >> issue. >> >> >> >> [1] htt
Re: Integration testing plugins
On 11 Aug 06, at 11:25 AM 11 Aug 06, John Casey wrote: it doesn't backup the whole local repository, only the files that will be affected by a plugin install. It should be relatively stable in terms of the backup/restore data transfer size...varying only by the number of repositories that were consulted for that artifact (because there is a set of metadata files for each remote repository). I'm not sure what you mean by encapsulating things needed by the runtime. What I want to avoid is introducing excessive design changes to the core just so we can enable plugin manipulation that wouldn't happen in a normal use case. I think this is a normal use case. A concrete example being the embedder pointing at a completely separate repository structure for, say, Geronimo. They use a Maven repository for their own use and you should be able to flip a parameter and use the encapsulated requirements for runtime: a repository, any configuration, any directories which hold configuration, caches, or anything of that nature. I think touching the repository that is actually used in the day-to-day is sort of fugly. If the plugin interacts with any artifacts then all those are going to be affected as so if you're doing integration tests on the Jetty plugin then you're going to pull in a whole raft of other dependencies, you're sure you're picking all that up and restoring it fully? I think some things with artifact resolution are a little loosy goosy at moment and not touching the users repository I think would be a better option. -john On 8/11/06, Jason van Zyl <[EMAIL PROTECTED]> wrote: On 11 Aug 06, at 9:18 AM 11 Aug 06, John Casey wrote: > This should be solved by the stage/unstage mojos in the plugin-test > plugin. > That's its only current purpose. It will stage a new plugin > artifact into > the local repository, backing up what was there before, then clear the > pluginManager's cache of that plugin, in case it's been resolved > already. > Unstaging (de-staging?) will restore the backed up data, and clear > the cache > again. > We should move toward encapsulating anything the runtime requires so we could parameterize it on execution so that you don't have to tamper with the local repository. If someone has a massive local repository that could get pretty time consuming. > -john > > On 8/11/06, Kenney Westerhof <[EMAIL PROTECTED]> wrote: >> >> >> >> Vincent Massol wrote: >> >> > Also, Kenney do you know if the it plugin can now be used to test a >> plugin >> > that is being built for the first time? I remember that I didn't >> activate it >> > on the clover plugin because it would work only if the clover >> plugin was >> > installed in the local repository and this came after integration >> testing. >> > Thus you would always run it tests on the previous version of >> the plugin >> > (looked up from the local repo). I think I had a jira issue for >> this. >> I'm >> > offline now but I'll try to dig it up if need be. >> >> The following issue [1] should provide that: it was blocking the >> use of >> the it plugin; once fixed, it worked fine. If the local repository >> version of the plugin is used instead, then I suggest we re- open the >> issue. >> >> [1] http://jira.codehaus.org/browse/MNG-870 >> >> -- Kenney >> >> - >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> >> Jason van Zyl [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] Jason van Zyl [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Integration testing plugins
it doesn't backup the whole local repository, only the files that will be affected by a plugin install. It should be relatively stable in terms of the backup/restore data transfer size...varying only by the number of repositories that were consulted for that artifact (because there is a set of metadata files for each remote repository). I'm not sure what you mean by encapsulating things needed by the runtime. What I want to avoid is introducing excessive design changes to the core just so we can enable plugin manipulation that wouldn't happen in a normal use case. -john On 8/11/06, Jason van Zyl <[EMAIL PROTECTED]> wrote: On 11 Aug 06, at 9:18 AM 11 Aug 06, John Casey wrote: > This should be solved by the stage/unstage mojos in the plugin-test > plugin. > That's its only current purpose. It will stage a new plugin > artifact into > the local repository, backing up what was there before, then clear the > pluginManager's cache of that plugin, in case it's been resolved > already. > Unstaging (de-staging?) will restore the backed up data, and clear > the cache > again. > We should move toward encapsulating anything the runtime requires so we could parameterize it on execution so that you don't have to tamper with the local repository. If someone has a massive local repository that could get pretty time consuming. > -john > > On 8/11/06, Kenney Westerhof <[EMAIL PROTECTED]> wrote: >> >> >> >> Vincent Massol wrote: >> >> > Also, Kenney do you know if the it plugin can now be used to test a >> plugin >> > that is being built for the first time? I remember that I didn't >> activate it >> > on the clover plugin because it would work only if the clover >> plugin was >> > installed in the local repository and this came after integration >> testing. >> > Thus you would always run it tests on the previous version of >> the plugin >> > (looked up from the local repo). I think I had a jira issue for >> this. >> I'm >> > offline now but I'll try to dig it up if need be. >> >> The following issue [1] should provide that: it was blocking the >> use of >> the it plugin; once fixed, it worked fine. If the local repository >> version of the plugin is used instead, then I suggest we re-open the >> issue. >> >> [1] http://jira.codehaus.org/browse/MNG-870 >> >> -- Kenney >> >> - >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> >> Jason van Zyl [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Integration testing plugins
On 11 Aug 06, at 9:18 AM 11 Aug 06, John Casey wrote: This should be solved by the stage/unstage mojos in the plugin-test plugin. That's its only current purpose. It will stage a new plugin artifact into the local repository, backing up what was there before, then clear the pluginManager's cache of that plugin, in case it's been resolved already. Unstaging (de-staging?) will restore the backed up data, and clear the cache again. We should move toward encapsulating anything the runtime requires so we could parameterize it on execution so that you don't have to tamper with the local repository. If someone has a massive local repository that could get pretty time consuming. -john On 8/11/06, Kenney Westerhof <[EMAIL PROTECTED]> wrote: Vincent Massol wrote: > Also, Kenney do you know if the it plugin can now be used to test a plugin > that is being built for the first time? I remember that I didn't activate it > on the clover plugin because it would work only if the clover plugin was > installed in the local repository and this came after integration testing. > Thus you would always run it tests on the previous version of the plugin > (looked up from the local repo). I think I had a jira issue for this. I'm > offline now but I'll try to dig it up if need be. The following issue [1] should provide that: it was blocking the use of the it plugin; once fixed, it worked fine. If the local repository version of the plugin is used instead, then I suggest we re-open the issue. [1] http://jira.codehaus.org/browse/MNG-870 -- Kenney - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] Jason van Zyl [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Integration testing plugins
This should be solved by the stage/unstage mojos in the plugin-test plugin. That's its only current purpose. It will stage a new plugin artifact into the local repository, backing up what was there before, then clear the pluginManager's cache of that plugin, in case it's been resolved already. Unstaging (de-staging?) will restore the backed up data, and clear the cache again. -john On 8/11/06, Kenney Westerhof <[EMAIL PROTECTED]> wrote: Vincent Massol wrote: > Also, Kenney do you know if the it plugin can now be used to test a plugin > that is being built for the first time? I remember that I didn't activate it > on the clover plugin because it would work only if the clover plugin was > installed in the local repository and this came after integration testing. > Thus you would always run it tests on the previous version of the plugin > (looked up from the local repo). I think I had a jira issue for this. I'm > offline now but I'll try to dig it up if need be. The following issue [1] should provide that: it was blocking the use of the it plugin; once fixed, it worked fine. If the local repository version of the plugin is used instead, then I suggest we re-open the issue. [1] http://jira.codehaus.org/browse/MNG-870 -- Kenney - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Integration testing plugins
Vincent Massol wrote: Also, Kenney do you know if the it plugin can now be used to test a plugin that is being built for the first time? I remember that I didn't activate it on the clover plugin because it would work only if the clover plugin was installed in the local repository and this came after integration testing. Thus you would always run it tests on the previous version of the plugin (looked up from the local repo). I think I had a jira issue for this. I'm offline now but I'll try to dig it up if need be. The following issue [1] should provide that: it was blocking the use of the it plugin; once fixed, it worked fine. If the local repository version of the plugin is used instead, then I suggest we re-open the issue. [1] http://jira.codehaus.org/browse/MNG-870 -- Kenney - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Integration testing plugins
> -Original Message- > From: Kenney Westerhof [mailto:[EMAIL PROTECTED] > Sent: vendredi 11 août 2006 00:13 > To: Maven Developers List > Subject: Re: Integration testing plugins > > > Hi John, > > It seems that the maven-invoker-plugin does actually the same thing > as the maven-it-plugin (still in the sandbox), except you added some > prebuild and verification steps. > > Maybe we can merge them somehow, or delete the maven-it-plugin (which I > actually wrote in the pre-embedder era and lots of it's code was moved > to the embedder). > > Do you think the maven-it-plugin is obsolete now? It runs a series of > builds on src/it/*/pom.xml, using the embedder. The idea was to have > unit tests in those it projects that tested plugins or other stuff (like > deployments). Cool stuff. Also in the same vein, does it make sense for the new maven-plugin-test-plugin (optionally) to use the verifier plugin to verify build results? Or maybe that's simply transparent as the verifier plugin is meant to be bound to the verify phase and is part of the build... The initial reason I wrote the verifier plugin was for plugin it testing in order to assert results from the build. I guess it would be nice to have a page somewhere on plugin testing explaining the different options: it plugin, plugin-test plugin, verifier plugin, test harness, any more? Also, Kenney do you know if the it plugin can now be used to test a plugin that is being built for the first time? I remember that I didn't activate it on the clover plugin because it would work only if the clover plugin was installed in the local repository and this came after integration testing. Thus you would always run it tests on the previous version of the plugin (looked up from the local repo). I think I had a jira issue for this. I'm offline now but I'll try to dig it up if need be. Thanks -Vincent > John Casey wrote: > > Hi, > > > > I wanted to let people know about something I've been working on for the > > assembly plugin. As anyone watching the commit notifications has > probably > > noticed, I've recently completed a large refactor of the assembly > plugin, > > with the aim of making it more modular and easier to unit test. While > this > > has made the source easier to comprehend and unit test, I've learned an > > important lesson in the process: test coverage numbers don't tell the > whole > > story. Some of you have tried to tell me this in the past, but let's > just > > say there's nothing like experiencing something firsthand... > > > > So, to make up for deficiencies in the unit tests (not checking every > > permutation of nulls, for example), I wanted to put together some > > functional > > tests which would help us set a high water mark for future releases, to > > prevent regressions (or try to). The trouble has been that this really > > requires running the plugin within its platform - i.e. Maven itself - > and > > checking the results of various builds. Further complicating things is > the > > fact that plugins are sometimes resolved early in the build lifecycle, > > meaning the instance that actually gets tested will be something > resolved > > from the local repository instead of the one you just built. > > > > This brings me to my announcement: I've written two new plugins, which > are > > currently in the sandbox, and which allow developers to run > > integration-test > > builds for a plugin. The way it works is relatively simple: > > > > 1. backup the local repository plugin directory structure, and stage out > > the > > fresh one > > 2. clear the resolved-plugin cache of the plugin you just built > > 3. run a series of Maven builds, executing pre-build and verification > > scripts before/after each build to setup and verify the test, > respectively > > 4. de-stage the plugin, meaning restore the local repository state prior > to > > the integration test setup > > > > For items 1, 2, and 4, I've written the maven-plugin-test-plugin (I > know, > > ugly name). Item 3 is another plugin I wrote, called the > > maven-invoker-plugin, which can be used to run any [set of] builds. It > uses > > beanshell scripts for prebuild and verification, can take CLI-injected > > properties from a test.properties file in the build directory, and > supports > > a goals.txt file for customizing the build command used per-project. > > > > As I mentioned, both of these plugins are in the sandbox. I've also > > deployed > > them to the apache.snapshots repository, and you can find a sample > > configurat
Re: Integration testing plugins
Yeah, I will follow up this thread with wiki documentation. I'd like to approach it as more of a "testing maven components" with a special section on plugins, personally...because I think a lot of the principles will apply in both cases. WDYT? Kenney: yeah, I actually revamped/gutted the it plugin for this one...it's dependent on the maven-invoker API that I developed in the open for a client. This API has a good set of tests, and is intentionally close to the embedder API so they can eventually be merged with an optional 'fork' flag. I wanted to use this plugin name rather than revamping the IT plugin, just in case there were plans for other things in the existing plugin...and also to escape the concept that this plugin is only for integration testing... ;-) I'd like to look at what features the IT plugin provides that this one doesn't, and maybe converging them where the behavior can be generalized as non-IT-specific. -j On 8/10/06, Brett Porter <[EMAIL PROTECTED]> wrote: cool. Can we get this written up in the wiki/docs like Jesse's stuff was? How about starting the "plugin developers centre" on the website? :) - Brett On 11/08/2006 8:06 AM, John Casey wrote: > Hi, > > I wanted to let people know about something I've been working on for the > assembly plugin. As anyone watching the commit notifications has probably > noticed, I've recently completed a large refactor of the assembly plugin, > with the aim of making it more modular and easier to unit test. While this > has made the source easier to comprehend and unit test, I've learned an > important lesson in the process: test coverage numbers don't tell the whole > story. Some of you have tried to tell me this in the past, but let's just > say there's nothing like experiencing something firsthand... > > So, to make up for deficiencies in the unit tests (not checking every > permutation of nulls, for example), I wanted to put together some > functional > tests which would help us set a high water mark for future releases, to > prevent regressions (or try to). The trouble has been that this really > requires running the plugin within its platform - i.e. Maven itself - and > checking the results of various builds. Further complicating things is the > fact that plugins are sometimes resolved early in the build lifecycle, > meaning the instance that actually gets tested will be something resolved > from the local repository instead of the one you just built. > > This brings me to my announcement: I've written two new plugins, which are > currently in the sandbox, and which allow developers to run > integration-test > builds for a plugin. The way it works is relatively simple: > > 1. backup the local repository plugin directory structure, and stage out > the > fresh one > 2. clear the resolved-plugin cache of the plugin you just built > 3. run a series of Maven builds, executing pre-build and verification > scripts before/after each build to setup and verify the test, respectively > 4. de-stage the plugin, meaning restore the local repository state prior to > the integration test setup > > For items 1, 2, and 4, I've written the maven-plugin-test-plugin (I know, > ugly name). Item 3 is another plugin I wrote, called the > maven-invoker-plugin, which can be used to run any [set of] builds. It uses > beanshell scripts for prebuild and verification, can take CLI-injected > properties from a test.properties file in the build directory, and supports > a goals.txt file for customizing the build command used per-project. > > As I mentioned, both of these plugins are in the sandbox. I've also > deployed > them to the apache.snapshots repository, and you can find a sample > configuration in a profile in the maven-assembly-plugin POM. > > Hopefully, we can use and evolve this pattern to start enabling > functional/integration tests for plugins which are easy to understand and > maintain. > > Cheers, > > John > -- Apache Maven - http://maven.apache.org/ Better Builds with Maven - http://library.mergere.com/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Integration testing plugins
cool. Can we get this written up in the wiki/docs like Jesse's stuff was? How about starting the "plugin developers centre" on the website? :) - Brett On 11/08/2006 8:06 AM, John Casey wrote: Hi, I wanted to let people know about something I've been working on for the assembly plugin. As anyone watching the commit notifications has probably noticed, I've recently completed a large refactor of the assembly plugin, with the aim of making it more modular and easier to unit test. While this has made the source easier to comprehend and unit test, I've learned an important lesson in the process: test coverage numbers don't tell the whole story. Some of you have tried to tell me this in the past, but let's just say there's nothing like experiencing something firsthand... So, to make up for deficiencies in the unit tests (not checking every permutation of nulls, for example), I wanted to put together some functional tests which would help us set a high water mark for future releases, to prevent regressions (or try to). The trouble has been that this really requires running the plugin within its platform - i.e. Maven itself - and checking the results of various builds. Further complicating things is the fact that plugins are sometimes resolved early in the build lifecycle, meaning the instance that actually gets tested will be something resolved from the local repository instead of the one you just built. This brings me to my announcement: I've written two new plugins, which are currently in the sandbox, and which allow developers to run integration-test builds for a plugin. The way it works is relatively simple: 1. backup the local repository plugin directory structure, and stage out the fresh one 2. clear the resolved-plugin cache of the plugin you just built 3. run a series of Maven builds, executing pre-build and verification scripts before/after each build to setup and verify the test, respectively 4. de-stage the plugin, meaning restore the local repository state prior to the integration test setup For items 1, 2, and 4, I've written the maven-plugin-test-plugin (I know, ugly name). Item 3 is another plugin I wrote, called the maven-invoker-plugin, which can be used to run any [set of] builds. It uses beanshell scripts for prebuild and verification, can take CLI-injected properties from a test.properties file in the build directory, and supports a goals.txt file for customizing the build command used per-project. As I mentioned, both of these plugins are in the sandbox. I've also deployed them to the apache.snapshots repository, and you can find a sample configuration in a profile in the maven-assembly-plugin POM. Hopefully, we can use and evolve this pattern to start enabling functional/integration tests for plugins which are easy to understand and maintain. Cheers, John -- Apache Maven - http://maven.apache.org/ Better Builds with Maven - http://library.mergere.com/ - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Integration testing plugins
Hi John, It seems that the maven-invoker-plugin does actually the same thing as the maven-it-plugin (still in the sandbox), except you added some prebuild and verification steps. Maybe we can merge them somehow, or delete the maven-it-plugin (which I actually wrote in the pre-embedder era and lots of it's code was moved to the embedder). Do you think the maven-it-plugin is obsolete now? It runs a series of builds on src/it/*/pom.xml, using the embedder. The idea was to have unit tests in those it projects that tested plugins or other stuff (like deployments). -- Kenney John Casey wrote: Hi, I wanted to let people know about something I've been working on for the assembly plugin. As anyone watching the commit notifications has probably noticed, I've recently completed a large refactor of the assembly plugin, with the aim of making it more modular and easier to unit test. While this has made the source easier to comprehend and unit test, I've learned an important lesson in the process: test coverage numbers don't tell the whole story. Some of you have tried to tell me this in the past, but let's just say there's nothing like experiencing something firsthand... So, to make up for deficiencies in the unit tests (not checking every permutation of nulls, for example), I wanted to put together some functional tests which would help us set a high water mark for future releases, to prevent regressions (or try to). The trouble has been that this really requires running the plugin within its platform - i.e. Maven itself - and checking the results of various builds. Further complicating things is the fact that plugins are sometimes resolved early in the build lifecycle, meaning the instance that actually gets tested will be something resolved from the local repository instead of the one you just built. This brings me to my announcement: I've written two new plugins, which are currently in the sandbox, and which allow developers to run integration-test builds for a plugin. The way it works is relatively simple: 1. backup the local repository plugin directory structure, and stage out the fresh one 2. clear the resolved-plugin cache of the plugin you just built 3. run a series of Maven builds, executing pre-build and verification scripts before/after each build to setup and verify the test, respectively 4. de-stage the plugin, meaning restore the local repository state prior to the integration test setup For items 1, 2, and 4, I've written the maven-plugin-test-plugin (I know, ugly name). Item 3 is another plugin I wrote, called the maven-invoker-plugin, which can be used to run any [set of] builds. It uses beanshell scripts for prebuild and verification, can take CLI-injected properties from a test.properties file in the build directory, and supports a goals.txt file for customizing the build command used per-project. As I mentioned, both of these plugins are in the sandbox. I've also deployed them to the apache.snapshots repository, and you can find a sample configuration in a profile in the maven-assembly-plugin POM. Hopefully, we can use and evolve this pattern to start enabling functional/integration tests for plugins which are easy to understand and maintain. Cheers, John - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
