Summarising this thread...
Terminology I mean when I mention things:
- unit testing, intended to test this module only, independent of its
interaction with Maven and dependencies
- integration testing, in this context, is intended to test that the
plugin works in Maven, that it's metadata is bound correctly, lifecycle
interactions, etc. Done via the embedder.
I saw this as being the unit testing side, and the reason we needed
stubs is because we weren't using real objects.
It sounds like what we have now is just great as it is, and I'd like to
start with that and see how far we get. My only issue remains the
confusing POM-that-isn't-a-POM that you were going to fix anyway.
I was concerned with the perceived direction towards using more real
Maven stuff. I think there are several levels to this:
* artifact code. I think it's quite reasonable to use this via
maven-artifact as it is fairly pojo like anyway. No new stubs needed.
* project builder. If you need a whole pom with interpolation and
inheritance, you build up a local repo and use the project builder. I
think this indicates a smell in the mojo (see more details below), but
we might have to live with it in the current setup. The framework
shouldn't care - it just takes in a MavenProject, and someone has
already either instantiated a stub, or built with the project builder
component. Still, we should start to consider the embedder here.
* artifact code that sets up the whole environment through
maven-artifact-manager. I dislike using this for mojos, though I set up
maven-artifact-test for it. Primarily for use by other libraries
(modello's tests, maybe artifact-ant). I think by this point we should
be thinking about the embedder anyway.
* the embedder should be used for anything that really relies on the
Maven environment, lifecycle handling, or any testing of how expressions
are set.
While the embedder certainly has its purposes, it should be used when
unit tests can't do something on their own.
I don't know if anyone has built the eclipse plugin lately, but here are
some figures:
* time to build skipping tests: 5 seconds
* time to build running tests: 25 seconds
* number of tests: 11
* number of (non-svn) files in src/test: 241
* code covered: ~65%
This really isn't scalable. More than 20 files per test and 2-3 seconds
each.
We now have 100 integration tests for Maven itself which take maybe 5-6
seconds each, much that could instead be done through unit tests. It's
just too slow to write and to run, and starting to defeat the purpose.
Bootstrapping is something you have to walk away from and grab a coffee
again.
So that's my summary of the situation. I'll review the framework and
tests in place and see where we go forward from here.
Specific responses....
Jesse McConnell wrote:
> the repo will need to be real at some point, some plugins needs to interact
> with the real thing, for getting jars for packaging or for just looking into
> for resources or something..so the artifact repo seems to be a required part
> of actual unit tests..
Probably, though it should never be a remote repo. There are already
some tools in maven-artifact-test for this.
I'm still very cautious about this though. It's really pushing into the
realms of integration testing.
> simple, I'll take care of this today... <plugin>-config.xml maybe?
I think it can be called whatever you like and passed in as a File,
right? If you look at some of the model tests, there is
pom-that-includes-something.xml, pom-that-doesnt.xml, etc.
> I think we can get away with all this without actually needing to add
> getters and setters, personally I think adding setters somewhat clouds the
> interaction with mojo's, have a combo of setters and private field injection
> seems prone to some kind of abuse
Well, plexus is using the setters if they are there anyway, so there's
no abuse. I'm fine with leaving that to the discretion of the Mojo
author. As long as its possible.
> I was getting visions of people just making one test pom and either reusing
> or copying and modifying a bit.
So say there are two fields with six possible values each and I need to
test them all with each other for some reason. Do I create 36 poms, or
do I create one and then call the Java setters to modify those values?
An example you can refer to here are the maven-project tests that test
scopes work from the POM. I felt like that was much harder to work with
than the equivalent unit tests later put into maven-artifact to do the
same thing.
> I agree with you here, it is disjointed to have the configuration and the
> asserts in another place...but if we are going to route of a new file for
> configuring these things, how about just building out an assert setup in the
> configuration xml?
>
> <test>
> <configuration>
> ...
> </configuration>
> <asserts>
> <assertTrue>expression</
> ..
> ..
Well, now you are really starting a separate testing framework and the
Java won't be required at all. That's a lot of work. Something I'd love
to investigate, but a lot of work. We probably should be looking at
using a scripting language to write the tests, perhaps.
Anyway... more responses.
Jesse McConnell wrote:
> Mojo's need some way of being tested and having coverage determined.
This actually works if the embedder is used via surefire. Not sure about
the IT plugin. The purpose is more about being tested easily, and quickly.
Jason van Zyl wrote:
> Brett Porter wrote:
>> A stub of the maven project builder, or the real one? The real one uses
>> the repo, and that's not really conducive to unit testing.
>
> The harness somewhat blurs the lines but once you execute the Mojo I
> don't really consider it a unit test anymore.
I don't see why it can't be. What happened to the OJO in MOJO?
> Even with the simple clean
> plugin example that you started with executed the Mojo. I think that's
> what's useful for testing and many Mojos need a local repository so I
> was suggesting using the real project builder passing in a local
> repository created for testing purposes.
The clean example is quirky because it basically does nothing outside of
what should be in external libraries, and has very little configuration.
If all of our plugins were like this, it would be great, and none of
this would be necessary, but there is plenty of Maven specific
functionality that needs to go into these if you look at plugins like
jar, eclipse, etc.
My problem with using a real project is that it is a really complicated
piece of code, that requires a local repository. Setting up that repo is
fine, but its really quite tedious. All of that complicated code runs
before the mojo is instantiated - so the mojo tests have no business
caring what it does. All we need is a prepopulated project object. Now,
we can certainly use the project builder for this, if it is simpler to
do so. I have no objections to it. But I like the idea of having a stub
that is a simple bean I can just call .setFoo() and .getFoo() on as well
on too.
Another reason I dislike the project builder is that we really should
not be using ${project} on its own in the mojos. That binds you to a
non-trivial library in Maven core that limits the ability to change the
version. More use of ${project.field} instead would be welcomed by me,
and I feel is more in the original spirit of Mojos.
As before, I think we need to start doing the tests and see where it
leads us. And definitely need to criticially review the tests and try
and find ways to make them simpler in code, and faster to write.
>> At one point, we were doing away with private field injection and
>> recommending setters, so it is how they'd be used.
>
> Hasn't happened yet. Not sure if that will happen as I'm not sure how
> many Ant tasks are actually used outside of Ant. Given what we do in
> promoting the creation of components that get wrapped by a Mojo to
> expose them in Maven I'm not sure if there's much point to properties.
> If that's what we decide then we can do that. I'm not fussed one way or
> the other.
Ok, likewise. I think this is tied into the dependence on Maven objects
instead of making them simple POJOs. Maybe it was putting the M in POJO
that was the problem, rather than a lack of OJO in MOJO :)
> But there are plugins that require a MavenProject and other parts of the
> POM not in the configuration for the plugin.
> The IDEA plugin is one example that needs a POM so having the pom.xml
> file actually be the source of a MavenProject I think would work. There
> is also the resources plugin which requires elements in the <build/>
> element. I don't think we'll get away with just the <configuration/>
> element for the plugin.
If it needs to be a POM, it should be a POM, loaded with the project
loader. If it is not a POM, it should not look like a POM, and just be
the XML configuration fragment inserted into the instantiated Mojo. Or
it could all be done in java code with a stub project and setters on the
mojo. I'd like the alternative for both.
> We've had this discussion before :-) They are not properties. We happen
> to abuse the field name being the same as the configuration element. But
> with a property the configuration element can be different then the
> field name as the setter would intervene.
Ok, I get confused on the terminology, sorry. But isn't it the case
right now that if I add a setter to a mojo, that gets called instead? Or
is that only if you add the property element to the configuration?
I think this needs to be written down in the documentation, as I suspect
nobody other than yourself knows how the setters truly work right now
and it's probably clouding our perception of how to deal with them here.
property="..." is not listed on
http://maven.apache.org/developers/mojo-api-specification.html.
> In how many cases in a Mojo can you only set one field and actually make
> it do anything?
surefirePlugin.setForkMode( "once" );
surefirePlugin.setForkMode( "pertest" );
surefirePlugin.setForkMode( "none" );
I think there are plenty of examples.
> I'm all for a Java way, but I think the pom.xml will be
> the most thorough and actually correspond most accurately to real use.
> Whatever kind of testing that actually is.
That's fine. Let's allow both, and find out through experience.
> In SuiteRunner the input and assertions were in the same file. Sure, it
> can't hurt to have both methods and I think we need a method in Java
> simply because we will probably find holes in the declarative structure
> and that shouldn't stop you from testing.
Ok, so we are generally in agreement. Summary at the top.
Cheers,
Brett
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
