Hi,
This is in response to John's email about integration testing, some
notes that I've taken while trying to clean up the integration
testing, and some suggestions about where we go from here with
respect to integration testing. You will see from the list below that
there are many problems with the ITs ranging from integration tests
for specific plugins being in there to using production dependencies,
to having tons of duplicated for doing ITs.
So, in response to John's email: I think we need to settle on what
we're going to use and stop writing new tools. We have several
invokers, several verifiers, several IT plugins, and the plugin
harness. It is simply out of control. We have different methods being
used in different plugins, nothing is standard and it's going to kill
us. The most prevalent tool, as defective as it might be is what is
being used in the ITs themselves. Stephane managed to use this
successfully in some of his plugins. Then after that we have an array
of usages. What should happen before we start writing more stuff is
to figure out something we can use now, and how to merge what we have
together instead of writing more tools.
This is just to get the discussion started. I will take a first pass
at enumerating the collection of tools we have and where they are
used. But here are my notes from the main core ITs. Thus far as some
of the ideas pertain to ITs in general like not polluting the local
repo when testing ..
Jason.
---
- [-] Goals
- [ ] An IT should be completely self-contained so that the
problem
can be understood by looking in one place, in one Maven
project.
- [ ] We should be able to create an Archetype so that users can
easily create ITs for us
- [+] The ITs should be in a project of their own so that we can
reuse them across versions of Maven. We could actually run
new versions of integration tests against old versions of
Maven. Solution: the ITs are now in a separate build
and it
is possible to run them
- [ ] We should be able to easily integrate the IT into a larger
run where we can use forked or embedded execution.
- [ ] We should create Archetypes for all categories of
problems so
that anyone can generate tests cases for us. Then there
is so
much that we can do in terms of automating this process of
checking tests for quality along with the patches.
- [ ] automate the testing of ITs submitted by users
- [ ] Each IT should have its own repository if it needs
resources
from repository. We can't mess with a users repository
when
testing.
- [ ] We need to have a file system based remote repository for
testing
- [ ] We need to standardize on integration testing in
general. We
have people going all over the place and it's a disaster.
- [ ] We have too many IT plugins (3)
- [ ] We have too many invokers (5)
- [ ] We have too many verifiers (3)
- [ ] The ITs should run nicely from an IDE. Solution: this does
work but requires that you run mvn clean
resources:testResources first as the IDE doesn't know
how to
set that up. Needs to be fully fixed. But it is much nicer
running this stuff in your IDE.
- [-] Problems with ITs
- [+] Verifier jar required by the bootstrap requires a special
verifier.jar there is no released version of this tool.
Solution: the bootstrap now uses Ant and we've gotten
rid of
a lot of the complexity.
- [+] The maven-core-it plugin needs to be decoupled into its
separate purposes because there are currently 12 different
things going on in the plugin and it would be really
confusing for a new user to figure out what's going on
in the
plugin and how it applies to the integration testing.
Solution: these have now been broken down into plugins
that
correspond to their function in the ITs.
- [ ] it0006 is an integration test for the verifier plugin.
- [ ] it0014 uses the compiler plugin to test plugin
configuration.
- [ ] it0016 is an integration test for the WAR plugin.
- [ ] it0017 is an integration test for the EJB plugin.
- [ ] it0018 uses real dependencies to test artifact resolution.
- [ ] it0020 tests beanshell mojo support.
- [ ] it0024 uses the compiler plugin to test mojo
configuration.
- [ ] it0028 uses the compiler plugin to test mojo
configuration.
- [ ] it0029 uses the compiler plugin to test for
pluginManagement
injection of plugin configuration.
- [ ] it0033 is an integration test for the EAR plugin.
- [ ] it0043 uses the help plugin to look at the state of the
final
POM, we should have this built into Maven for testing.
- [ ] it0047 uses plexus-container-default for testing.
- [ ] it0048 uses the surefire plugin to test default value
population for mojo parameters.
- [ ] it0050 is an integration test for the surefire plugin.
- [ ] it0051 is an integration test for the release plugin.
- [ ] it0052 is an integration test for the release plugin.
- [ ] it0054 is an integration test for the resources plugin.
- [ ] it0055 is an integration test for the compiler plugin.
- [ ] it0056 uses the compiler plugin to test multiple
executions.
- [ ] it0060 uses the compiler plugin to test aggregation of
list
configuration items when using 'combine.children=append'
attribute.
- [ ] it0063 tests JDK 1.4.2 specifics.
- [ ] it0068 uses the modello plugin to test repository
accumulation (i'm not sure what that means, John?)
- [ ] it0069 uses classworlds dependency to test offline mode.
- [ ] it0070 is an integration test for the RAR plugin.
- [ ] it0074 uses the compiler plugin to test mojo
configuration.
- [ ] it0076 uses the WAR plugin to test pluginManagement.
- [ ] it0078 uses the compiler plugin to test pluginManagement.
- [ ] it0079 uses the source plugin to test that attached
artifacts
have the same build number as the primary artifact.
- [ ] it0080 uses the WAR plugin to test an artifact handler
that
specifies no contribution to transitive dependencies.
if the
behavior of this handler changed the test would break.
- [ ] it0081 uses production dependency of wagon as a dependency
- [ ] it0083 uses the WAR plugin to test an artifact handler
that
the WAR artifact handler currently behaves like
- [ ] it0086 uses production dependency of bsh to test
classloading
- [ ] it0087 uses prodcution dependency of commons-logging to
test
classloading
- [ ] it0089 is an integration test for the checkstyle plugin
- [ ] it0094 has a test that attempts to access a protected
field
in a ClassRealm
- [ ] it0095 uses the help and verifier plugins to test URL
calculation
- [ ] it0100 uses the antrun plugin to make sure
${parent.artifactId} resolves correctly
- [ ] it0102 uses the help and antrun plugins to test profile
activation
- [ ] it0104 uses the surefire plugin to test interpolation
- [ ] it0105 is an integration test for the resources plugin
- [ ] artifactIds should be aligned with directories
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
