On 04/01/2007, at 4:32 PM, Brett Porter wrote:
Jason - any further thoughts on this?
Ping... No is a valid answer :)
I'd like to get your summary put somewhere individuals can pick
things off to work on - probably a jira project for shared. WDYT?
I'm overcommitted for working on things right now, but I know a
couple of people are confused about the IT testing, and we've got all
those chronically broken plugins. Volunteers?
On 18/12/2006, at 4:35 PM, Brett Porter wrote:
On 12/12/2006, at 5:08 AM, Jason van Zyl wrote:
So, in response to John's email: I think we need to settle on
what we're going to use and stop writing new tools. We have
several invokers, several verifiers, several IT plugins, and the
plugin harness. It is simply out of control. We have different
methods being used in different plugins, nothing is standard and
it's going to kill us. The most prevalent tool, as defective as
it might be is what is being used in the ITs themselves. Stephane
managed to use this successfully in some of his plugins. Then
after that we have an array of usages. What should happen before
we start writing more stuff is to figure out something we can use
now, and how to merge what we have together instead of writing
more tools.
Agreed. I think that's what John was getting at too, but by doing
it clean rather than rewriting something that was in use
somewhere. So the work left to do, in either case, is apply it
consistently and get rid of the stuff we aren't going to use.
To me, it looks like:
- the plugin-testing-harness needs to go. They should be
integration tests that use a proper pom, or use pure mocks rather
than the stubs that tend to just have a bunch of impossible-to-get-
under-real-condition values.
- John's test tools have the most complete invocation options, and
tools for managing repositories that we can reuse, so I'd opt for
that in that area
- the verifier is well utilised, so if that is merged with the
code from the verifier plugin then we can lose the invocation
stuff and repository management stuff and merge it all together
Can we have a wiki page with this work list? Or, can we check in
the omni outliner files + an export for the non-mac users to review?
[snip points I agree with]
- [+] The ITs should be in a project of their own so that we can
reuse them across versions of Maven. We could actually run
new versions of integration tests against old versions of
Maven. Solution: the ITs are now in a separate build
and it
is possible to run them
How should this play out in plugins? I would be in favour of
separate projects for these.
Two things that are must haves for me:
- integration tests / anything that forks a Maven instance must
*not* be part of the normal test build for a plugin. They take way
too long, and lead to use of maven.test.skip :(
- as far as integration testing in general, I think we recommend a
separate project, but enable them to be part of the same project
for simplicity of development (this is more specific to things
like web applications where it is probably beneficial).
- [ ] We should be able to easily integrate the IT into a larger
run where we can use forked or embedded execution.
Not sure what you mean here. What is an IT that *doesn't* use
forked or embedded execution?
- [ ] automate the testing of ITs submitted by users
what does this mean? I think, like a patch, a submitted IT still
needs to be reviewed, and incorporated into the main test suite
(with corresponding fix-for version so it only runs when it is
expected to work). Otherwise, long term, we'll have lots of
duplicated or poorly conceived ITs. I've seen test cases submitted
that are quite useful at demonstrating something, but contain half
of the user's proect which is not good for our long term scalability.
- [ ] Each IT should have its own repository if it needs
resources
from repository. We can't mess with a users repository
when
testing.
- [ ] We need to have a file system based remote repository for
testing
Agreed. Isn't that what John's tool already does?
- [ ] We need to standardize on integration testing in
general. We
have people going all over the place and it's a disaster.
- [ ] We have too many IT plugins (3)
- [ ] We have too many invokers (5)
- [ ] We have too many verifiers (3)
Let's specifically get these mapped out and a path forward so that
everyone can push towards it, rather than relying on you to do the
work.
- [ ] The ITs should run nicely from an IDE. Solution: this does
work but requires that you run mvn clean
resources:testResources first as the IDE doesn't know
how to
set that up. Needs to be fully fixed. But it is much nicer
running this stuff in your IDE.
Agree with the point, but not sure what you are referring to about
testResources - the generated projects for IDEA and I think
Eclipse already do this (and obviously better integration will
bring it).
I think the dangerous thing is using resources for non-classpath
resources. It's better for the tests to setup and use a clean
project instance for an IT itself (using helper tools).
[snip specific notes on old ITs that need to be updated]
- [ ] artifactIds should be aligned with directories
Agreed. Also, as I did in the last test I wrote, what do we think
about using purposeful names rather than numbers for integration
tests?
Thanks for this.
Cheers,
Brett
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
