Re: Invoker vs. Verifier?
On Dec 12, 2007, at 9:47 PM, Dan Fabulich wrote: I tweak the test to add a MAVEN_OPTS environment variable, including the -Xrunjdwp string. (It would be easy to add some sugar to Verifier and/or Invoker to make this easier; I didn't want to go fooling around with the Verifier, so I just copied and pasted out of my notes when I needed this.) Do you have an actual example of this that I can look at? As for the rest, I'll be interested to see what you come up with. I definitely agree that this Tower of Babel approach to things that we have in place now isn't helpful. I'm still not completely convinced that it's effective to do all of this through the IDE (yet), but I'm definitely willing to see an example of how it can be improved, as long as this doesn't mean compromising readability or debug-ability of headless build logs, as in the case of a test failing on the CI server, but not on my localhost. --- John Casey Committer and PMC Member, Apache Maven mail: jdcasey at commonjava dot org blog: http://www.ejlife.net/blogs/john rss: http://feeds.feedburner.com/ejlife/john
Re: Invoker vs. Verifier?
John Casey wrote: On Dec 12, 2007, at 9:47 PM, Dan Fabulich wrote: I tweak the test to add a MAVEN_OPTS environment variable, including the -Xrunjdwp string. (It would be easy to add some sugar to Verifier and/or Invoker to make this easier; I didn't want to go fooling around with the Verifier, so I just copied and pasted out of my notes when I needed this.) Do you have an actual example of this that I can look at? In Eclipse/IDEA you can launch the test and modify the test configuration to include different environment variables; it's all GUI and hard to provide an example. However, you can also do it by modifying the code of the test like this: Verifier verifier = new Verifier( testDir.getAbsolutePath() ); HashMap envVars = new HashMap(); envVars.put( MAVEN_OPTS, -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8001 -Xnoagent -Djava.compiler=NONE ); verifier.executeGoal( test , envVars ); verifier.verifyErrorFreeLog(); verifier.resetStreams(); -Dan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Invoker vs. Verifier?
Hi, IMHO, don't use maven-embedder. Because for a plugin with a required maven version = 2.0.6, tests with embedder will failed. Look at the jar plugin all tests has been migrated to maven-invoker-plugin due to this. -- Olivier 2007/12/12, Johan Kindgren [EMAIL PROTECTED]: I created a few integration-tests for the jar-plugin using the Maven Embedder as described in http://maven.apache.org/developers/committer-testing-plugins.html under both maven-it-plugin and maven-plugin-test-plugin (I didn't use any testing plugins, just the install-plugin). Add the Embedder to your list of testing utilities. I think a lot of things could be improved in the testing strategies. The bugs that I fixed in the jar-plugin took me less than an hour to complete, but creating the testcases to me about three days before I finally found the above mentioned page. Maybe I've missunderstood something completely, but as far as I understand some tests are auctually testing the result from the previous build? /Johan 2007/12/12, Brian E. Fox [EMAIL PROTECTED]: You forgot the maven-plugin-testing-harness ;-) -Original Message- From: Dan Fabulich [mailto:[EMAIL PROTECTED] Sent: Tuesday, December 11, 2007 9:07 PM To: Maven Developers List Subject: Re: Invoker vs. Verifier? John Casey wrote: What you're seeing as overlap is a mixture of concerns in the invoker plugin. The verifications beanshell really needs to be migrated out to some sort of proper integration-testing plugin (or, even better, a plugin that unites invoker and verifier under a common configuration...then extend the verifier with the invoker's beanshell functionality). Regardless, the invoker plugin can be used for any sort of scenario where you need to fork a new maven process. I've personally used it to proxy secondary builds in some sticky client use cases. You don't have to use the beanshell script to verify the build, it's just an [admittedly confusing] option. As I've remarked before, I find it weird that various Maven developers have gone and written _plugins_ to do Maven integration testing. Integration tests are just tests; we know how to write/run tests using real test frameworks like JUnit and TestNG. Those frameworks are pretty cool; you can do stuff like rerun failures-only, graph results over time, write data-driven tests, etc. You can even use them to write tests in scripting languages like Groovy, BeanShell, etc. All that AND you get excellent IDE integration. More generally, while I certainly see the value of a maven-invoker-plugin, I don't expect that you'd want that to be the normal way people would write Maven integration tests. Right now there are four things: maven-verifier, maven-verifier-plugin (no relation!), maven-invoker, and maven-invoker-plugin. I think I'd like to advocate ripping out the bulk of maven-verifier and make it depend entirely on maven-invoker. Since maven-verifier is so confusingly named, I think I'd want to take the good bits out and put them in maven-integration-test-helper (which is what maven-verifier really is, anyway). More controversially (?) I'd like to deprecate the idea of writing *tests* using the maven-invoker-plugin, instead preferring to write them in Java (or BeanShell, I'm easy!) running them using a real test framework. maven-invoker-plugin should still be used for spawning sub-builds in those delightful cases where that's necessary. Thoughts? -Dan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Johan Kindgren Acrend AB Phone: +46 (0) 733-58 36 60 E-mail: [EMAIL PROTECTED] www.acrend.se - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Invoker vs. Verifier?
Dan Fabulich wrote: John Casey wrote: What you're seeing as overlap is a mixture of concerns in the invoker plugin. The verifications beanshell really needs to be migrated out to some sort of proper integration-testing plugin (or, even better, a plugin that unites invoker and verifier under a common configuration...then extend the verifier with the invoker's beanshell functionality). Regardless, the invoker plugin can be used for any sort of scenario where you need to fork a new maven process. I've personally used it to proxy secondary builds in some sticky client use cases. You don't have to use the beanshell script to verify the build, it's just an [admittedly confusing] option. As I've remarked before, I find it weird that various Maven developers have gone and written _plugins_ to do Maven integration testing. Integration tests are just tests; we know how to write/run tests using real test frameworks like JUnit and TestNG. Those frameworks are pretty cool; you can do stuff like rerun failures-only, graph results over time, write data-driven tests, etc. You can even use them to write tests in scripting languages like Groovy, BeanShell, etc. All that AND you get excellent IDE integration. More generally, while I certainly see the value of a maven-invoker-plugin, I don't expect that you'd want that to be the normal way people would write Maven integration tests. Right now there are four things: maven-verifier, maven-verifier-plugin (no relation!), maven-invoker, and maven-invoker-plugin. I think I'd like to advocate ripping out the bulk of maven-verifier and make it depend entirely on maven-invoker. Since maven-verifier is so confusingly named, I think I'd want to take the good bits out and put them in maven-integration-test-helper (which is what maven-verifier really is, anyway). More controversially (?) I'd like to deprecate the idea of writing *tests* using the maven-invoker-plugin, instead preferring to write them in Java (or BeanShell, I'm easy!) running them using a real test framework. maven-invoker-plugin should still be used for spawning sub-builds in those delightful cases where that's necessary. Thoughts? Dan, I agree with you - integration tests would benefit from having a more unified approach, just like unit tests are run by surefire. It should support different languages, Java and scripting, and different runners. I would still have a different plugin for ITs than for unit tests, as they typically are run with different usecases, and I fear that surefire would get overloaded. Cheers - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Invoker vs. Verifier?
First things first. maven-invoker and maven-invoker-plugin are not separate things. The maven-invoker-plugin uses maven-invoker, but maven-invoker is meant to be a reusable library, not just a plugin. Aside from this, there is actually a good reason for running integration tests using maven and maven plugins. Many users can submit stripped-down project builds that exhibit a certain error; fewer would be able (or willing) to submit the same failure in the form of a junit test. A failing build should be able to be incorporated directly into the test suite, and if it makes sense, later be split into lower-level junit tests to check the behavior of a certain component. JUnit is best suited to unit tests, as its name suggests. For integration testing, where you actually need to integrate the container (Maven, I mean) with a suite of plugins, IMO it makes much more sense to provide the flexibility of executing maven as a separate process (or embedded, this option simply hasn't been developed for the invoker plugin yet...mainly because the 2.0.x embedder sucks). This makes it trivial to test command-line options, different versions of maven, or any number of configurations that are somewhat external to Maven's core apis, without having to develop and constantly verify the validity of testing harnesses. The maven- verifier (not the verifier plugin, I know it's confusing) gets away from this, in that it requires a project directory in src/test/ resources AND a JUnit test case to orchestrate the test. IMO, this makes it extremely difficult to run a single core integration test from the command line, so you can capture logs for offline examination, for instance. Again, the idea behind test builds driven by the invoker is to provide a test suite with a low barrier to entry, and with all the potential for configuration present in the main user's experience with Maven. It's not a panacea, but neither is writing JUnit tests that orchestrate and verify test-build resources in a separate directory structure, which can really only be run properly from the JUnit api. Another thing to keep in mind is that you have two different integration-testing profiles to deal with, and they are distinct: 1. testing the core features built into maven 2. testing the functionality of a single plugin in different situations I'm not at all sure that it makes sense to try and merge these two scenarios behind a single api...what would it give you? I do think the verifier used in core-integration-tests should be merged with some other, more general verification mechanism. However, it's also critical to allow a project's POM to remain unmodified for the purposes of orchestrating or verifying the build. Modifying the POM introduces the potential to taint the test results, and can limit flexibility in terms of verification. For instance, if you need to simply scan the console output for a particular message, it becomes much more difficult to do this if you try to do it while the build is still running. Just to be clear, I don't think writing invoker-plugin-driven test builds is a good substitute for writing unit tests. Unit tests will cover the code far more completely in most cases, and as you pointed out, can be graphed and rendered into all sorts of shiny reports much more readily. Also, I'd like to know how you would suggest writing integration tests using junit without the invoker functionality (the real meat of the invoker plugin) and without access to the maven 2.1 embedder (it's not available in any real way in 2.0.x). I guess what I'm wondering is how these tests will really look, once you've taken out the verifier and eliminated the use of the invoker (for the most part, at least)? You say JUnit or even Beanshell, but those are just technologies; how will you use them? -john On Dec 11, 2007, at 10:06 PM, Dan Fabulich wrote: John Casey wrote: What you're seeing as overlap is a mixture of concerns in the invoker plugin. The verifications beanshell really needs to be migrated out to some sort of proper integration-testing plugin (or, even better, a plugin that unites invoker and verifier under a common configuration...then extend the verifier with the invoker's beanshell functionality). Regardless, the invoker plugin can be used for any sort of scenario where you need to fork a new maven process. I've personally used it to proxy secondary builds in some sticky client use cases. You don't have to use the beanshell script to verify the build, it's just an [admittedly confusing] option. As I've remarked before, I find it weird that various Maven developers have gone and written _plugins_ to do Maven integration testing. Integration tests are just tests; we know how to write/run tests using real test frameworks like JUnit and TestNG. Those frameworks are pretty cool; you can do stuff like
Re: Invoker vs. Verifier?
John Casey wrote: First things first. maven-invoker and maven-invoker-plugin are not separate things. The maven-invoker-plugin uses maven-invoker, but maven-invoker is meant to be a reusable library, not just a plugin. I find this remark quite confusing... if one is a library, and one is a plugin, and they have two separate POMs, one for the library JAR (maven-invoker) and one for the plugin (maven-invoker-plugin) doesn't that make them two things that share code? Actually, I think our disagreement is mostly a misunderstanding, and I think it mostly turns on this point of whether maven-invoker and maven-invoker plugin are two things. In particular, I said that we should not use maven-invoker-plugin for integration testing. But I didn't mean that we shouldn't use maven-invoker! e.g. when you said: Also, I'd like to know how you would suggest writing integration tests using junit without the invoker functionality (the real meat of the invoker plugin) and without access to the maven 2.1 embedder (it's not available in any real way in 2.0.x). I never meant to suggest that we'd write integration tests without the invoker. I meant that maven-verifier should *use* maven-invoker (the library) and should continue to execute a separate instance of Maven. We're in total agreement that integration testing should happen by running real Maven projects and executing a real Maven to make that happen, for all the reasons you laid out. [I also argued that we should rename maven-verifier, because its name is confusing.] With that said, I think there is still a point of actual disagreement lurking here which I'd like to flush out... namely whether it's best to write your tests in Java/JUnit/TestNG or whether you'd want to have/use/write a test-runner plugin (beyond Surefire that just runs JUnit/TestNG) to handle these tests. More generally, I think plugin integration test writers should write their tests in the maven-verifier style (using a maven invocation library [e.g. maven-invoker] under a JUnit test), and not using maven-invoker-plugin. I think that because both tests of Maven Core and integration tests of a plugin call for doing the same thing: invoking Maven and asserting on the results. JUnit is best suited to unit tests, as its name suggests. True, though JUnit is acceptable for integration tests too, (especially JUnit 4) and TestNG is great for both. The two of them together beat the pants off of almost everything else in Java land. The maven-verifier (not the verifier plugin, I know it's confusing) gets away from this, in that it requires a project directory in src/test/resources AND a JUnit test case to orchestrate the test. IMO, this makes it extremely difficult to run a single core integration test from the command line, so you can capture logs for offline examination, for instance. With that, I definitely disagree. Can you say more about what's difficult about running a single test? Certainly it's trivial to run a single JUnit test from the command line using Surefire: mvn test -Dtest=FooTest does the job nicely; I use that all the time. And it's REALLY easy to run a single test from Eclipse/IDEA. Maybe you meant that you think maven-verifier tests are harder to WRITE than writing goals.txt + beanshell tests? (I disagree with that, too, but it's worth clarifying what we're talking about.) Again, the idea behind test builds driven by the invoker is to provide a test suite with a low barrier to entry, and with all the potential for configuration present in the main user's experience with Maven. It's not a panacea, but neither is writing JUnit tests that orchestrate and verify test-build resources in a separate directory structure, which can really only be run properly from the JUnit api. You call it the JUnit API like it's this big hassle to run JUnit tests... but it's really easy to run JUnit tests, both from the command line and from an IDE; certainly 99% of our users know how to do it, and I hope most of them do it frequently! :-) Furthermore, you seem to imply here that you can't just cd into src/test/resources and run the projects there by hand directly... but of course you can do that when you want/need to. I often do that when I'm first setting up an integration test, before I've written the JUnit assertions for that test. Occasionally I do that when a test fails just so I can make sure I can reproduce manually what the test is doing automatically. However, it's also critical to allow a project's POM to remain unmodified for the purposes of orchestrating or verifying the build. Modifying the POM introduces the potential to taint the test results, and can limit flexibility in terms of verification. For instance, if you need to simply scan the console output for a particular message, it becomes much more difficult to do this if you try to do it while the build is still running. I don't understand
Re: Invoker vs. Verifier?
On Dec 12, 2007, at 4:18 PM, Dan Fabulich wrote: John Casey wrote: First things first. maven-invoker and maven-invoker-plugin are not separate things. The maven-invoker-plugin uses maven-invoker, but maven-invoker is meant to be a reusable library, not just a plugin. I find this remark quite confusing... if one is a library, and one is a plugin, and they have two separate POMs, one for the library JAR (maven-invoker) and one for the plugin (maven-invoker- plugin) doesn't that make them two things that share code? Actually, I think our disagreement is mostly a misunderstanding, and I think it mostly turns on this point of whether maven-invoker and maven-invoker plugin are two things. In particular, I said that we should not use maven-invoker-plugin for integration testing. But I didn't mean that we shouldn't use maven-invoker! To me, shared code implies some sort of mutual dependency on something, whether that's cyclical or dependent on the same third library. Sorry for the confusion. Having said that, I definitely disagree with the approach you outline below. See my comments inline. The maven-verifier (not the verifier plugin, I know it's confusing) gets away from this, in that it requires a project directory in src/test/resources AND a JUnit test case to orchestrate the test. IMO, this makes it extremely difficult to run a single core integration test from the command line, so you can capture logs for offline examination, for instance. With that, I definitely disagree. Can you say more about what's difficult about running a single test? Certainly it's trivial to run a single JUnit test from the command line using Surefire: mvn test -Dtest=FooTest does the job nicely; I use that all the time. And it's REALLY easy to run a single test from Eclipse/IDEA. Maybe you meant that you think maven-verifier tests are harder to WRITE than writing goals.txt + beanshell tests? (I disagree with that, too, but it's worth clarifying what we're talking about.) Not at all; I mean running the test. In order to run one of these tests (which are orchestrated by something akin to the maven-verifier from a JUnit or other java-driven test case), you must run JUnit or whatever, so you can be sure you have the same options enabled, environment present, and assertions tested as take place in the test- case code. For instance, simply cd into src/test/resources/it0105 in core-integration-tests, and see if you can figure out what it tests, and how. You can't, not without looking at the JUnit code that drives it, to determine what the criteria for success is, and which flags and goals/lifecycle phases to invoke. If I needed to re-run this test to actually diagnose a failure (which is the whole point here), I have to dig around in source code that's completely outside the it0105 directory, then come back and replicate that maven command, with the addition of the -X flag and a pipe to the tee command so I can analyze the output outside of the build. This is much harder than it needs to be, and the same is true for plugin integration tests. Again, the idea behind test builds driven by the invoker is to provide a test suite with a low barrier to entry, and with all the potential for configuration present in the main user's experience with Maven. It's not a panacea, but neither is writing JUnit tests that orchestrate and verify test-build resources in a separate directory structure, which can really only be run properly from the JUnit api. You call it the JUnit API like it's this big hassle to run JUnit tests... but it's really easy to run JUnit tests, both from the command line and from an IDE; certainly 99% of our users know how to do it, and I hope most of them do it frequently! :-) Furthermore, you seem to imply here that you can't just cd into src/ test/resources and run the projects there by hand directly... but of course you can do that when you want/need to. I often do that when I'm first setting up an integration test, before I've written the JUnit assertions for that test. Occasionally I do that when a test fails just so I can make sure I can reproduce manually what the test is doing automatically. Again, it's not just about running the tests, but being able to actually debug a failing test effectively. Tests work best when they're easy to understand and work with, and when a maven core- integration-test fails, you can definitely see how this setup falls down. Running and re-running the same test without change from the IDE isn't useful for debugging, and running the invoker from this kind of code with the remote debugging enabled is virtually impossible...incidentally, if you've figured out how to do it, I'd be interested in learning. However, it's also critical to allow a project's POM to remain unmodified for the purposes of orchestrating or verifying the
Re: Invoker vs. Verifier?
John Casey wrote: Not at all; I mean running the test. In order to run one of these tests (which are orchestrated by something akin to the maven-verifier from a JUnit or other java-driven test case), you must run JUnit or whatever, so you can be sure you have the same options enabled, environment present, and assertions tested as take place in the test-case code. I stand by my claim that running JUnit is very easy. For instance, simply cd into src/test/resources/it0105 in core-integration-tests, and see if you can figure out what it tests, and how. You can't, not without looking at the JUnit code that drives it, to determine what the criteria for success is, and which flags and goals/lifecycle phases to invoke. I have two remarks about this. 1) Don't do it that way, and 2) We can compromise on the directory structure. 1) Don't do it that way: I think this is a very weird way to approach the tests that only seems natural because people are used to doing it that way. I'll try to explain what I mean with a hypothetical story. Suppose you were testing some XML serializer/deserializer, and you had a bunch of JUnit tests checked into src/test/java and a bunch of XML files checked into src/test/resources. If you're going to go browsing around to see what kinds of tests are available, where are you going to go first? I'd argue that the most natural thing to do is to go look at the tests themselves in src/test/java. The XML files aren't tests... they're just test RESOURCES. Looking at the tests is a good idea for another reason also: the tests can (must) have some reference in their code to the resources they're using, telling you (and the JVM!) where to find the resources. The resources, on the other hand, don't have to (and typically won't) have a reference to the tests that will run them. Similarly, Maven projects aren't tests. They may be test resources. It's weird to go browsing around the test resources trying to figure out what tests might run them. Instead, you should be looking at the tests themselves. Indeed, looking at the tests it's very easy to see what resources they're using. Even in the maven-invoker-plugin case, you shouldn't go digging around in the project-under-test *first*. First you should look at the commands/assertions in the POM; see what it's doing. THEN go look at the project-under-test. It's the same thing: look at the test first, not the resource. 2) We can compromise the directory structure: Despite those remarks, sure, it would be nicer if you could have your JUnit/TestNG integration test in the same directory as its own test resources. That's true of all JUnit/TestNG tests by the way, not just Maven integration tests. The Maven standard is to keep test resources in src/test/resources, separate from the tests in src/test/java. But it doesn't have to be that way... lots of people put their resources in the same directory as the classes that use them. You could, if you wished, just put them all together into one directory, claiming that this directory is both the testSourceDirectory AND the testResource directory. (I think you'd want some exclusion/inclusion rules to keep everything straight, but that's not so bad.) With that said, I think that one should resist the temptation to do this in all but the simplest cases. What about integration tests that test multiple projects? You can't put the same test in multiple directories. Or what about projects that are used by multiple integration tests? The relationship of tests to resources can be many-to-many. I claim that the clearest thing is to follow the Maven standard, even if it's sub-optimal in this case: keep your resources in one directory and have your tests refer to them. Ultimately, this part of the argument is about which directory structure is clearest. As we know, arguments about clarity of directory structure can be highly controversial, but are never conclusive and don't amount to very much. Nobody wins arguments about directory structure; everyone's a loser. If you buy my argument that it's good to write tests in a real test framework in a normal language, that writing tests in a POM is not ideal, then I'm pretty sure we can deal with any remaining concerns about directory structure. I'm easy! :-) Again, it's not just about running the tests, but being able to actually debug a failing test effectively. Tests work best when they're easy to understand and work with, and when a maven core-integration-test fails, you can definitely see how this setup falls down. Running and re-running the same test without change from the IDE isn't useful for debugging, and running the invoker from this kind of code with the remote debugging enabled is virtually impossible...incidentally, if you've figured out how to do it, I'd be interested in learning. I wrote a lot of new integration tests in the maven-verifier style for Surefire,
Re: Invoker vs. Verifier?
What you're seeing as overlap is a mixture of concerns in the invoker plugin. The verifications beanshell really needs to be migrated out to some sort of proper integration-testing plugin (or, even better, a plugin that unites invoker and verifier under a common configuration...then extend the verifier with the invoker's beanshell functionality). Regardless, the invoker plugin can be used for any sort of scenario where you need to fork a new maven process. I've personally used it to proxy secondary builds in some sticky client use cases. You don't have to use the beanshell script to verify the build, it's just an [admittedly confusing] option. -john On Dec 11, 2007, at 6:28 PM, Dan Fabulich wrote: Forgive me if I'm picking at a sore spot, but can someone help me understand the difference/overlap between maven-invoker and maven- verifier? As I understand it, they both do roughly the same thing, except one of them is a Maven plugin where you write your test in a goals.txt file + beanshell, and the other is meant to be run programmatically from a JUnit test. Is that essentially correct? (Whereas maven-verifier-plugin is another beast entirely, simply checking for the presence/absence of certian files.) Am I right in thinking that, despite their substantial overlap, maven-verifier and maven-invoker share no code? -Dan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --- John Casey Committer and PMC Member, Apache Maven mail: jdcasey at commonjava dot org blog: http://www.ejlife.net/blogs/john rss: http://feeds.feedburner.com/ejlife/john
Re: Invoker vs. Verifier?
John Casey wrote: What you're seeing as overlap is a mixture of concerns in the invoker plugin. The verifications beanshell really needs to be migrated out to some sort of proper integration-testing plugin (or, even better, a plugin that unites invoker and verifier under a common configuration...then extend the verifier with the invoker's beanshell functionality). Regardless, the invoker plugin can be used for any sort of scenario where you need to fork a new maven process. I've personally used it to proxy secondary builds in some sticky client use cases. You don't have to use the beanshell script to verify the build, it's just an [admittedly confusing] option. As I've remarked before, I find it weird that various Maven developers have gone and written _plugins_ to do Maven integration testing. Integration tests are just tests; we know how to write/run tests using real test frameworks like JUnit and TestNG. Those frameworks are pretty cool; you can do stuff like rerun failures-only, graph results over time, write data-driven tests, etc. You can even use them to write tests in scripting languages like Groovy, BeanShell, etc. All that AND you get excellent IDE integration. More generally, while I certainly see the value of a maven-invoker-plugin, I don't expect that you'd want that to be the normal way people would write Maven integration tests. Right now there are four things: maven-verifier, maven-verifier-plugin (no relation!), maven-invoker, and maven-invoker-plugin. I think I'd like to advocate ripping out the bulk of maven-verifier and make it depend entirely on maven-invoker. Since maven-verifier is so confusingly named, I think I'd want to take the good bits out and put them in maven-integration-test-helper (which is what maven-verifier really is, anyway). More controversially (?) I'd like to deprecate the idea of writing *tests* using the maven-invoker-plugin, instead preferring to write them in Java (or BeanShell, I'm easy!) running them using a real test framework. maven-invoker-plugin should still be used for spawning sub-builds in those delightful cases where that's necessary. Thoughts? -Dan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Invoker vs. Verifier?
You forgot the maven-plugin-testing-harness ;-) -Original Message- From: Dan Fabulich [mailto:[EMAIL PROTECTED] Sent: Tuesday, December 11, 2007 9:07 PM To: Maven Developers List Subject: Re: Invoker vs. Verifier? John Casey wrote: What you're seeing as overlap is a mixture of concerns in the invoker plugin. The verifications beanshell really needs to be migrated out to some sort of proper integration-testing plugin (or, even better, a plugin that unites invoker and verifier under a common configuration...then extend the verifier with the invoker's beanshell functionality). Regardless, the invoker plugin can be used for any sort of scenario where you need to fork a new maven process. I've personally used it to proxy secondary builds in some sticky client use cases. You don't have to use the beanshell script to verify the build, it's just an [admittedly confusing] option. As I've remarked before, I find it weird that various Maven developers have gone and written _plugins_ to do Maven integration testing. Integration tests are just tests; we know how to write/run tests using real test frameworks like JUnit and TestNG. Those frameworks are pretty cool; you can do stuff like rerun failures-only, graph results over time, write data-driven tests, etc. You can even use them to write tests in scripting languages like Groovy, BeanShell, etc. All that AND you get excellent IDE integration. More generally, while I certainly see the value of a maven-invoker-plugin, I don't expect that you'd want that to be the normal way people would write Maven integration tests. Right now there are four things: maven-verifier, maven-verifier-plugin (no relation!), maven-invoker, and maven-invoker-plugin. I think I'd like to advocate ripping out the bulk of maven-verifier and make it depend entirely on maven-invoker. Since maven-verifier is so confusingly named, I think I'd want to take the good bits out and put them in maven-integration-test-helper (which is what maven-verifier really is, anyway). More controversially (?) I'd like to deprecate the idea of writing *tests* using the maven-invoker-plugin, instead preferring to write them in Java (or BeanShell, I'm easy!) running them using a real test framework. maven-invoker-plugin should still be used for spawning sub-builds in those delightful cases where that's necessary. Thoughts? -Dan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Invoker vs. Verifier?
I created a few integration-tests for the jar-plugin using the Maven Embedder as described in http://maven.apache.org/developers/committer-testing-plugins.html under both maven-it-plugin and maven-plugin-test-plugin (I didn't use any testing plugins, just the install-plugin). Add the Embedder to your list of testing utilities. I think a lot of things could be improved in the testing strategies. The bugs that I fixed in the jar-plugin took me less than an hour to complete, but creating the testcases to me about three days before I finally found the above mentioned page. Maybe I've missunderstood something completely, but as far as I understand some tests are auctually testing the result from the previous build? /Johan 2007/12/12, Brian E. Fox [EMAIL PROTECTED]: You forgot the maven-plugin-testing-harness ;-) -Original Message- From: Dan Fabulich [mailto:[EMAIL PROTECTED] Sent: Tuesday, December 11, 2007 9:07 PM To: Maven Developers List Subject: Re: Invoker vs. Verifier? John Casey wrote: What you're seeing as overlap is a mixture of concerns in the invoker plugin. The verifications beanshell really needs to be migrated out to some sort of proper integration-testing plugin (or, even better, a plugin that unites invoker and verifier under a common configuration...then extend the verifier with the invoker's beanshell functionality). Regardless, the invoker plugin can be used for any sort of scenario where you need to fork a new maven process. I've personally used it to proxy secondary builds in some sticky client use cases. You don't have to use the beanshell script to verify the build, it's just an [admittedly confusing] option. As I've remarked before, I find it weird that various Maven developers have gone and written _plugins_ to do Maven integration testing. Integration tests are just tests; we know how to write/run tests using real test frameworks like JUnit and TestNG. Those frameworks are pretty cool; you can do stuff like rerun failures-only, graph results over time, write data-driven tests, etc. You can even use them to write tests in scripting languages like Groovy, BeanShell, etc. All that AND you get excellent IDE integration. More generally, while I certainly see the value of a maven-invoker-plugin, I don't expect that you'd want that to be the normal way people would write Maven integration tests. Right now there are four things: maven-verifier, maven-verifier-plugin (no relation!), maven-invoker, and maven-invoker-plugin. I think I'd like to advocate ripping out the bulk of maven-verifier and make it depend entirely on maven-invoker. Since maven-verifier is so confusingly named, I think I'd want to take the good bits out and put them in maven-integration-test-helper (which is what maven-verifier really is, anyway). More controversially (?) I'd like to deprecate the idea of writing *tests* using the maven-invoker-plugin, instead preferring to write them in Java (or BeanShell, I'm easy!) running them using a real test framework. maven-invoker-plugin should still be used for spawning sub-builds in those delightful cases where that's necessary. Thoughts? -Dan - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Johan Kindgren Acrend AB Phone: +46 (0) 733-58 36 60 E-mail: [EMAIL PROTECTED] www.acrend.se - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
