On 27/12/2007, at 4:15 PM, Dan Fabulich wrote:
1) This document draws a distinction between "integration testing"
and "functional testing." The two terms are defined, but I don't
think I quite understand the distinction in practice.
In particular, this doc states that only two of the available
solutions "can be used for integration testing:" the maven-it-plugin
"ITP" and the maven-verifier-plugin "VRP."
I find that puzzling, because VRP does VERY little: it just verifies
that a file exists and (optionally) that its content matches a
regex. It's hard to see how it could be used in anything but a
functional test, in which you'd run a full Maven build and then
assert on the output...?
You're right - typo. (I did make quite a few :)
As for maven-it-plugin, I didn't think maven-it-plugin could do a
different kind of testing from what maven-invoker-plugin does...
There's even a remark to that effect in the Notes section of your
wiki article: "IT plugin is now duplicated with invoker."
What does it mean when this doc says that maven-it-plugin can do
integration testing but maven-invoker-plugin can't?
You're right on this too - I wasn't really thinking straight about it.
Let me elaborate on what I was getting at.
Integration testing can legitimately take a small subset of modules to
integrate or all of them so it has a pretty broad definition. I've
generally thought of the Maven ITs to be functional/system tests
rather than integration tests. What I was looking for is something
integrated, but only used in the context of the thing being tested
(the plugin). So you'd use the embedder to construct the system as
usual, but then pre-set the state and only execute the portion of the
lifecycle that runs the mojo. The example I was thinking of was
testing source generation - there's really no need to compile the
sources later on to verify that source generation worked as you
expected, and it may be a time consuming thing better reserved for a
system test.
Anyway, I've updated the table and re-labeled "contextual integration
testing" (if that's even the right terminology) and indicated there is
no current solution for it, and made the integration testing results
more accurate.
2) Three of the solutions have the feature "Runs with Surefire:"
maven-verifier (VER), maven-plugin-testing-harness (PTH), and "maven-
component-it-plugin with Java wrappers generated" (CIJ). In those
cases where they run under Surefire, I would think that a number of
the categories listed would be N/A.
For example, "able to run/timeout tests in parallel", "can ignore
failures", and "can skip execution" are pretty clearly Surefire
features. Weirdly, CIJ marks this feature as N/A, PTH gets a (/) and
VER gets a (x). But surely all three of these should have the same
value, right?
For those, I think it's pretty clear that all three of them should
get a (/). But consider ANSI color support. Surefire doesn't
support that right now (I just filed it as SUREFIRE-420), so I guess
those tools that use Surefire would get a (x). But if we do wind up
supporting ANSI color some day, then they all get a (/), without
ever touching the old plugins.
As for Groovy support, you can write Groovy tests and run them under
Surefire right now (I just tried it); Surefire just runs classes in
test-classes, ignorant of how they got there. Any language that
compiles to .class and extends TestCase should work, including
Jython and Rhino-compiled JavaScript. More generally, anything that
runs in JUnit will run under Surefire.
I think that means that CIJ, VER and PTH all support writing tests
in Groovy, right?
Yeah, the medium for capturing the information wasn't particularly
helpful here, but some consistency would have helped :)
For Surefire - I've added a flag for features that they derive from
Surefire. For Groovy - I've clarified that this was WRT to just the
setup/verify scripts.
I've also made some updates to the structure of the table that should
help better clarify and tried to make it more consistent - does it help?
3) What is "able to run a regex set of tests?" Is that just running
a specific subset of tests defined by a regex? e.g. if you had
tests FooBar, FooBaz, and BlahBar, you could run just "Foo.*" or
just ".*Bar"?
Do Surefire's wildcard-based includes count? (You know you can even
run them from the command line, with -Dtest=Foo* ...)
If not, should we introduce a new feature called "able to run a
wildcard set of tests"?
I didn't know Surefire supported that :) Have renamed regex to
wildcard/regex and update.
4) What is "interpolate goals/properties to run Maven with?"
the goals file can contain expressions (maybe John can explain why
it's useful, I was just going through the feature list of the invoker
plugin).
5) What is "Provides alternate remote repository handling?"
Apparently nothing has it. What would it do if someone did support
it?
Something Brian wanted in one of the mail threads. I think what he
wanted was to be able to have Maven behave like it would talking to a
remote repo, without having to talk to a remote repo, and without
having to do a lot of legwork to set up a file based one to test with?
I've moved it to the unit testing specific section.
Finally, I've got some quibbling. By my count, VER gets 39 (x)s.
Of those, I asked questions about 5 of them ("integration" testing,
regexes, interpolation of goals/properties, alternate remote repo
handling), 16 I'd let stand, and I'd like to quibble about the other
19. :-)
6) These three I think are just mistakes... they clearly work today.
* Suite teardown steps (public void tearDown)
fixed
* Cleans up logs, targets (verifier does this...?)
it doesn't remove things from /tmp I believe - I clarified the text to
include that
* Configurable working directory (that's the only argument to
Verifier's constructor!)
fixed
7) These five should all be (/), because Surefire supports them:
* Able to run tests in parallel
* Able to timeout tests run in parallel
* Can ignore failures
* Can skip execution
* Groovy support
fixed. I guess the groovy one could be either way - verifier doesn't
support using it for it's setup/teardown (it has it's own format), but
you could write them into the unit test itself...
8) VER got six (x)s that all amount to roughly the same thing:
controlling the flags that get passed to the invoked Maven, whether
JVM flags or command line arguments.
* Configurable individual Maven settings
* Pass profiles to run Maven with
* Pass profiles to run Maven with in a file
* Pass flags to run Maven with
* Pass flags to run Maven with in a file
* Able to run Maven in debug mode
True, updated.
9) Can be used for report plugin testing: I test the surefire-report
plugin using VER; it verifies that the report ran successfully
without failing the Maven build. You can't verify a report
completely without also adding HtmlUnit (which I haven't bothered to
do), but HtmlUnit is definitely compatible with Verifier. That's at
least as good as maven-invoker-plugin, which got a (/).
Updated.
10) "Isolated from user's environment/settings" and "Able to inherit
shell's environment:" In fact, Verifier inherits the shell
environment by default, which is what makes it less likely to be
isolated from the user's environment. Still, you CAN deliberately
isolate yourself further, if you wish, by specifying your
environment variables when you call executeGoal, and by passing in
an explicit Maven settings file.
We should split this into "Isolated from user's environment/settings
by default" and "Can isolate from user's environment/settings." And
"Able to inherit shell environment" should be (/).
Done.
11) Can select which projects to run (include/exclude). Projects
are just resources. Tests are what you run. This should be N/A or
(/); you can certainly select which tests to run (include/exclude)
using Surefire.
Done.
12) Able to drop-in an IT easily from a user submission. I don't
think this is specific enough. I think VER is pretty easy; other
people apparently disagree. This should be phrased as "able to drop
in an IT without _______" where "_______" is something hard.
That, of course, requires us to agree on what "______" should be and
whether it's hard. As I've argued in the past, some people have
argued that "_____" is "writing a JUnit test," which I insist is
very easy.
Took it out... I agree we could add some more specific stuff.
Cheers,
Brett
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
