Jason - any further thoughts on this?
On 18/12/2006, at 4:35 PM, Brett Porter wrote:
On 12/12/2006, at 5:08 AM, Jason van Zyl wrote:
So, in response to John's email: I think we need to settle on what
we're going to use and stop writing new tools. We have several
invokers, several verifiers, several IT plugins, and the plugin
harness. It is simply out of control. We have different methods
being used in different plugins, nothing is standard and it's
going to kill us. The most prevalent tool, as defective as it
might be is what is being used in the ITs themselves. Stephane
managed to use this successfully in some of his plugins. Then
after that we have an array of usages. What should happen before
we start writing more stuff is to figure out something we can use
now, and how to merge what we have together instead of writing
more tools.
Agreed. I think that's what John was getting at too, but by doing
it clean rather than rewriting something that was in use somewhere.
So the work left to do, in either case, is apply it consistently
and get rid of the stuff we aren't going to use.
To me, it looks like:
- the plugin-testing-harness needs to go. They should be
integration tests that use a proper pom, or use pure mocks rather
than the stubs that tend to just have a bunch of impossible-to-get-
under-real-condition values.
- John's test tools have the most complete invocation options, and
tools for managing repositories that we can reuse, so I'd opt for
that in that area
- the verifier is well utilised, so if that is merged with the code
from the verifier plugin then we can lose the invocation stuff and
repository management stuff and merge it all together
Can we have a wiki page with this work list? Or, can we check in
the omni outliner files + an export for the non-mac users to review?
[snip points I agree with]
- [+] The ITs should be in a project of their own so that we can
reuse them across versions of Maven. We could actually run
new versions of integration tests against old versions of
Maven. Solution: the ITs are now in a separate build and it
is possible to run them
How should this play out in plugins? I would be in favour of
separate projects for these.
Two things that are must haves for me:
- integration tests / anything that forks a Maven instance must
*not* be part of the normal test build for a plugin. They take way
too long, and lead to use of maven.test.skip :(
- as far as integration testing in general, I think we recommend a
separate project, but enable them to be part of the same project
for simplicity of development (this is more specific to things like
web applications where it is probably beneficial).
- [ ] We should be able to easily integrate the IT into a larger
run where we can use forked or embedded execution.
Not sure what you mean here. What is an IT that *doesn't* use
forked or embedded execution?
- [ ] automate the testing of ITs submitted by users
what does this mean? I think, like a patch, a submitted IT still
needs to be reviewed, and incorporated into the main test suite
(with corresponding fix-for version so it only runs when it is
expected to work). Otherwise, long term, we'll have lots of
duplicated or poorly conceived ITs. I've seen test cases submitted
that are quite useful at demonstrating something, but contain half
of the user's proect which is not good for our long term scalability.
- [ ] Each IT should have its own repository if it needs
resources
from repository. We can't mess with a users repository when
testing.
- [ ] We need to have a file system based remote repository for
testing
Agreed. Isn't that what John's tool already does?
- [ ] We need to standardize on integration testing in
general. We
have people going all over the place and it's a disaster.
- [ ] We have too many IT plugins (3)
- [ ] We have too many invokers (5)
- [ ] We have too many verifiers (3)
Let's specifically get these mapped out and a path forward so that
everyone can push towards it, rather than relying on you to do the
work.
- [ ] The ITs should run nicely from an IDE. Solution: this does
work but requires that you run mvn clean
resources:testResources first as the IDE doesn't know
how to
set that up. Needs to be fully fixed. But it is much nicer
running this stuff in your IDE.
Agree with the point, but not sure what you are referring to about
testResources - the generated projects for IDEA and I think Eclipse
already do this (and obviously better integration will bring it).
I think the dangerous thing is using resources for non-classpath
resources. It's better for the tests to setup and use a clean
project instance for an IT itself (using helper tools).
[snip specific notes on old ITs that need to be updated]
- [ ] artifactIds should be aligned with directories
Agreed. Also, as I did in the last test I wrote, what do we think
about using purposeful names rather than numbers for integration
tests?
Thanks for this.
Cheers,
Brett
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
