First things first. maven-invoker and maven-invoker-plugin are not
separate things. The maven-invoker-plugin uses maven-invoker, but
maven-invoker is meant to be a reusable library, not just a plugin.
Aside from this, there is actually a good reason for running
integration tests using maven and maven plugins. Many users can
submit stripped-down project builds that exhibit a certain error;
fewer would be able (or willing) to submit the same failure in the
form of a junit test. A failing build should be able to be
incorporated directly into the test suite, and if it makes sense,
later be split into lower-level junit tests to check the behavior of
a certain component. JUnit is best suited to unit tests, as its name
suggests. For integration testing, where you actually need to
integrate the container (Maven, I mean) with a suite of plugins, IMO
it makes much more sense to provide the flexibility of executing
maven as a separate process (or embedded, this option simply hasn't
been developed for the invoker plugin yet...mainly because the 2.0.x
embedder sucks). This makes it trivial to test command-line options,
different versions of maven, or any number of configurations that are
somewhat external to Maven's core apis, without having to develop and
constantly verify the validity of testing harnesses. The maven-
verifier (not the verifier plugin, I know it's confusing) gets away
from this, in that it requires a project directory in src/test/
resources AND a JUnit test case to orchestrate the test. IMO, this
makes it extremely difficult to run a single core integration test
from the command line, so you can capture logs for offline
examination, for instance.
Again, the idea behind test builds driven by the invoker is to
provide a test suite with a low barrier to entry, and with all the
potential for configuration present in the main user's experience
with Maven. It's not a panacea, but neither is writing JUnit tests
that orchestrate and verify test-build resources in a separate
directory structure, which can really only be run properly from the
JUnit api.
Another thing to keep in mind is that you have two different
integration-testing profiles to deal with, and they are distinct:
1. testing the core features built into maven
2. testing the functionality of a single plugin in different situations
I'm not at all sure that it makes sense to try and merge these two
scenarios behind a single api...what would it give you? I do think
the verifier used in core-integration-tests should be merged with
some other, more general verification mechanism. However, it's also
critical to allow a project's POM to remain unmodified for the
purposes of orchestrating or verifying the build. Modifying the POM
introduces the potential to taint the test results, and can limit
flexibility in terms of verification. For instance, if you need to
simply scan the console output for a particular message, it becomes
much more difficult to do this if you try to do it while the build is
still running.
Just to be clear, I don't think writing invoker-plugin-driven test
builds is a good substitute for writing unit tests. Unit tests will
cover the code far more completely in most cases, and as you pointed
out, can be graphed and rendered into all sorts of shiny reports much
more readily.
Also, I'd like to know how you would suggest writing integration
tests using junit without the invoker functionality (the real meat of
the invoker plugin) and without access to the maven 2.1 embedder
(it's not available in any real way in 2.0.x).
I guess what I'm wondering is how these tests will really look, once
you've taken out the verifier and eliminated the use of the invoker
(for the most part, at least)? You say JUnit or even Beanshell, but
those are just technologies; how will you use them?
-john
On Dec 11, 2007, at 10:06 PM, Dan Fabulich wrote:
John Casey wrote:
What you're seeing as overlap is a mixture of concerns in the
invoker plugin. The verifications beanshell really needs to be
migrated out to some sort of proper integration-testing plugin
(or, even better, a plugin that unites invoker and verifier under
a common configuration...then extend the verifier with the
invoker's beanshell functionality). Regardless, the invoker plugin
can be used for any sort of scenario where you need to fork a new
maven process. I've personally used it to proxy secondary builds
in some sticky client use cases. You don't have to use the
beanshell script to verify the build, it's just an [admittedly
confusing] option.
As I've remarked before, I find it weird that various Maven
developers have gone and written _plugins_ to do Maven integration
testing.
Integration tests are just tests; we know how to write/run tests
using real test frameworks like JUnit and TestNG. Those frameworks
are pretty cool; you can do stuff like rerun failures-only, graph
results over time, write data-driven tests, etc. You can even use
them to write tests in scripting languages like Groovy, BeanShell,
etc. All that AND you get excellent IDE integration.
More generally, while I certainly see the value of a maven-invoker-
plugin, I don't expect that you'd want that to be the "normal" way
people would write Maven integration tests.
Right now there are four things: maven-verifier, maven-verifier-
plugin (no relation!), maven-invoker, and maven-invoker-plugin.
I think I'd like to advocate ripping out the bulk of maven-verifier
and make it depend entirely on maven-invoker. Since maven-verifier
is so confusingly named, I think I'd want to take the good bits out
and put them in maven-integration-test-helper (which is what maven-
verifier really is, anyway).
More controversially (?) I'd like to deprecate the idea of writing
*tests* using the maven-invoker-plugin, instead preferring to write
them in Java (or BeanShell, I'm easy!) running them using a "real"
test framework. maven-invoker-plugin should still be used for
spawning sub-builds in those delightful cases where that's necessary.
Thoughts?
-Dan
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---
John Casey
Committer and PMC Member, Apache Maven
mail: jdcasey at commonjava dot org
blog: http://www.ejlife.net/blogs/john
rss: http://feeds.feedburner.com/ejlife/john
