I agree that branching probably is maintenance hell.
In the old setup we had a file that listed all the tests and we could append
each line with list of version
ranges to indicate the maven version the test was for. In the new setup, this
is not possible.
Some thoughts:
- Annotating the integration tests within Junit/TestNG test code makes it
harder to manually
run a test on a non-supported maven version (for instance to check if some
test is a regression,
or wheter it works for some version; we'd also need to modify all test code
when a new version
is put out. Ideally the testcases never change, only new ones are added.
- In the new setup (last time I looked) the testcases were defined by java classes.
We could have some metadata present in the pom files; this makes it easier to maintain than
specifying it in the test code itself.
On the other hand, you can't get a good overview of what tests are available
for which versions of maven.
Specifying that in a separate file (instead of test code or test poms) solves
this, but this could create
more maintenance requirements.
- If we have one top level integration test svn project (with
trunk,branches,tags), you can't test maven-version specific features.
For instance, the plugin tools: if you want to test wheter the parameterType
mojo annotation (present in 2.1, not in 2.0.x),
you're bound to use some specific version of a plugin or dependency. This
won't accurately represent the maven
version being tested, or, even worse, the version is ignored since the
dependency is in $M2_HOME/lib.
So this limits the tests you can perform for integration testing. You can
still (and should, IMHO) provide (integration) tests
in the project containing the new feature itself.
I think we'll have to reconsider what tests are global integration tests and
which ones arent:
* run for multiple versions of maven?
* tests complex featuresets, where lots of things can go wrong (for instance
in a multiproject build: inheritance, interpolation,
reactor project ordering/resolving, remote/local repository, reactor
artifact resolution, dependencies). All of these features
are ofcourse tested separately, but the global integration tests should test
wheter these work together properly
* one integration test for each Jira issue, if the bug indicates something
complex enough to warrant an integration test.
-- Kenney
Jason van Zyl wrote:
On 30 Oct 06, at 2:33 AM 30 Oct 06, Dan Fabulich wrote:
Jason and I had discussed this briefly a while ago... In my opinion,
the best/clearest way to handle this is to branch the integration tests.
Yes, we had the discussion but I don't agree that branching them is the
best idea. I think that's just going to be a maintenance burden and I
don't see that as a long term scalable solution. If we end up over time
having N versions of Maven I do not want to maintain N branches of
integration tests.
I think using something like TestNG we can easily create groups based
versions using annotations, or pick off attributes in the POMs in
addition to attributes in the tests.
I also think that having one standard base of tests makes everyone think
about all versions that we purport to actually support, test, and maintain.
When we are done we could easily package our ITs up into a single
deliverable so that we can let people run them on our machine to gather
even more feedback from the field. The single set of ITs is actually
sitting with me quite when and I like it more everyday I play around
with them. I think it will be quite easy using TestNG to pick of a test
for a particular environment and I can easily rig something up to check
the POM for <prerequisite/> in the existing setup.
What drawbacks do you see with having a single set of tests for all
branches? Also consider that we want to use the same setup for plugins,
which Stephan has tried for the EAR plugin, and I can't really imagine
having N branches for each plugin. I think that would just get out of
control very fast. I think we have a much greater chance of making
something truly maintainable using this setup.
Jason.
-Dan
-----Original Message-----
From: Brett Porter [mailto:[EMAIL PROTECTED]
Sent: Sunday, October 29, 2006 4:24 PM
To: Maven Developers List
Subject: integration tests: how to handle versions?
Hi,
I like the new integration testing method. However, I've just added a
test that will only work with a version of Maven that fixes the bug
(which I have sorted out locally, just doing some other checks before
committing).
So the two things we need to do:
- exclude that test from Maven < 2.0.5
- display what Maven version is being used before starting, because I
keep accidentally using the wrong version.
Maybe the <prerequisite /> tag in the pom.xml for the test (and
handle the error case from Maven), or just have a condition in the
test that if mavenVersion < 2.0.5(which would need to be obtained in
the verifier, perhaps by scraping mvn -version), output SKIPPED.
Any thoughts on the best way to do this?
- Brett
_______________________________________________________________________
Notice: This email message, together with any attachments, may contain
information of BEA Systems, Inc., its subsidiaries and affiliated
entities, that may be confidential, proprietary, copyrighted and/or
legally privileged, and is intended solely for the use of the individual
or entity named in this message. If you are not the intended recipient,
and have received this message in error, please immediately return this
by email and then delete it.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
