its back, the thread from heck and gone... ok, so now that we have a moderately working plugin harness that I have been able to get working for a few plugins, what should be the next step on this?
Talking it over with jdcasey we both loathe the idea of just moving the plugin harness directly over to the plugin trunk. One idea that we both kinda liked was pulling a branch and further developing the plugin harness and the test cases for each plugin there and then when we are happy just doing a wholesale merge to the trunk. This shouldn't be bad as I'll illustrate next :) The nature of the harness means that having to modify the actual plugin source in any way is an indication that we missed something (IMO at least) so we shouldn't have to worry about breaking plugins at least up to the test phase...Adding tests to the plugin should only be a matter of preparing a *MojoTest and the various plugin-pom.xml files for each test case in a directory structure. For the examples I was working with I did: src/test/resources/unit/<shortDesc>-test/plugin-pom.xml then in the test case when I looked up my mojo I used the location that the file would have been copied to in: target/test-classes/unit/<shortDesc>-test/plugin-pom.xml an other files that were needed were nested below the plugin-pom.xml in src/main/foo or whatever. so...any ideas on how this ought to go from here? oh, and the jira issue on this and related confluence documentation is: http://jira.codehaus.org/browse/MNG-32 http://docs.codehaus.org/display/MAVEN/Maven+Plugin+Harness Jesse On 3/7/06, Vincent Massol <[EMAIL PROTECTED]> wrote: > > +1 to all that! Exactly my sentiment. > > Thanks > -Vincent > > > -----Original Message----- > > From: Brett Porter [mailto:[EMAIL PROTECTED] > > Sent: mardi 7 mars 2006 22:37 > > To: Maven Developers List > > Subject: Re: plugin testing > > > > Summarising this thread... > > > > Terminology I mean when I mention things: > > - unit testing, intended to test this module only, independent of its > > interaction with Maven and dependencies > > - integration testing, in this context, is intended to test that the > > plugin works in Maven, that it's metadata is bound correctly, lifecycle > > interactions, etc. Done via the embedder. > > > > I saw this as being the unit testing side, and the reason we needed > > stubs is because we weren't using real objects. > > > > It sounds like what we have now is just great as it is, and I'd like to > > start with that and see how far we get. My only issue remains the > > confusing POM-that-isn't-a-POM that you were going to fix anyway. > > > > I was concerned with the perceived direction towards using more real > > Maven stuff. I think there are several levels to this: > > > > * artifact code. I think it's quite reasonable to use this via > > maven-artifact as it is fairly pojo like anyway. No new stubs needed. > > > > * project builder. If you need a whole pom with interpolation and > > inheritance, you build up a local repo and use the project builder. I > > think this indicates a smell in the mojo (see more details below), but > > we might have to live with it in the current setup. The framework > > shouldn't care - it just takes in a MavenProject, and someone has > > already either instantiated a stub, or built with the project builder > > component. Still, we should start to consider the embedder here. > > > > * artifact code that sets up the whole environment through > > maven-artifact-manager. I dislike using this for mojos, though I set up > > maven-artifact-test for it. Primarily for use by other libraries > > (modello's tests, maybe artifact-ant). I think by this point we should > > be thinking about the embedder anyway. > > > > * the embedder should be used for anything that really relies on the > > Maven environment, lifecycle handling, or any testing of how expressions > > are set. > > > > While the embedder certainly has its purposes, it should be used when > > unit tests can't do something on their own. > > > > I don't know if anyone has built the eclipse plugin lately, but here are > > some figures: > > * time to build skipping tests: 5 seconds > > * time to build running tests: 25 seconds > > * number of tests: 11 > > * number of (non-svn) files in src/test: 241 > > * code covered: ~65% > > > > This really isn't scalable. More than 20 files per test and 2-3 seconds > > each. > > > > We now have 100 integration tests for Maven itself which take maybe 5-6 > > seconds each, much that could instead be done through unit tests. It's > > just too slow to write and to run, and starting to defeat the purpose. > > Bootstrapping is something you have to walk away from and grab a coffee > > again. > > > > So that's my summary of the situation. I'll review the framework and > > tests in place and see where we go forward from here. > > > > Specific responses.... > > > > Jesse McConnell wrote: > > > the repo will need to be real at some point, some plugins needs to > > interact > > > with the real thing, for getting jars for packaging or for just > looking > > into > > > for resources or something..so the artifact repo seems to be a > required > > part > > > of actual unit tests.. > > > > Probably, though it should never be a remote repo. There are already > > some tools in maven-artifact-test for this. > > > > I'm still very cautious about this though. It's really pushing into the > > realms of integration testing. > > > > > simple, I'll take care of this today... <plugin>-config.xml maybe? > > > > I think it can be called whatever you like and passed in as a File, > > right? If you look at some of the model tests, there is > > pom-that-includes-something.xml, pom-that-doesnt.xml, etc. > > > > > I think we can get away with all this without actually needing to add > > > getters and setters, personally I think adding setters somewhat clouds > > the > > > interaction with mojo's, have a combo of setters and private field > > injection > > > seems prone to some kind of abuse > > > > Well, plexus is using the setters if they are there anyway, so there's > > no abuse. I'm fine with leaving that to the discretion of the Mojo > > author. As long as its possible. > > > > > I was getting visions of people just making one test pom and either > > reusing > > > or copying and modifying a bit. > > > > So say there are two fields with six possible values each and I need to > > test them all with each other for some reason. Do I create 36 poms, or > > do I create one and then call the Java setters to modify those values? > > > > An example you can refer to here are the maven-project tests that test > > scopes work from the POM. I felt like that was much harder to work with > > than the equivalent unit tests later put into maven-artifact to do the > > same thing. > > > > > I agree with you here, it is disjointed to have the configuration and > > the > > > asserts in another place...but if we are going to route of a new file > > for > > > configuring these things, how about just building out an assert setup > in > > the > > > configuration xml? > > > > > > <test> > > > <configuration> > > > ... > > > </configuration> > > > <asserts> > > > <assertTrue>expression</ > > > .. > > > .. > > > > Well, now you are really starting a separate testing framework and the > > Java won't be required at all. That's a lot of work. Something I'd love > > to investigate, but a lot of work. We probably should be looking at > > using a scripting language to write the tests, perhaps. > > > > Anyway... more responses. > > > > Jesse McConnell wrote: > > > Mojo's need some way of being tested and having coverage determined. > > > > This actually works if the embedder is used via surefire. Not sure about > > the IT plugin. The purpose is more about being tested easily, and > quickly. > > > > Jason van Zyl wrote: > > > Brett Porter wrote: > > >> A stub of the maven project builder, or the real one? The real one > uses > > >> the repo, and that's not really conducive to unit testing. > > > > > > The harness somewhat blurs the lines but once you execute the Mojo I > > > don't really consider it a unit test anymore. > > > > I don't see why it can't be. What happened to the OJO in MOJO? > > > > > Even with the simple clean > > > plugin example that you started with executed the Mojo. I think that's > > > what's useful for testing and many Mojos need a local repository so I > > > was suggesting using the real project builder passing in a local > > > repository created for testing purposes. > > > > The clean example is quirky because it basically does nothing outside of > > what should be in external libraries, and has very little configuration. > > If all of our plugins were like this, it would be great, and none of > > this would be necessary, but there is plenty of Maven specific > > functionality that needs to go into these if you look at plugins like > > jar, eclipse, etc. > > > > My problem with using a real project is that it is a really complicated > > piece of code, that requires a local repository. Setting up that repo is > > fine, but its really quite tedious. All of that complicated code runs > > before the mojo is instantiated - so the mojo tests have no business > > caring what it does. All we need is a prepopulated project object. Now, > > we can certainly use the project builder for this, if it is simpler to > > do so. I have no objections to it. But I like the idea of having a stub > > that is a simple bean I can just call .setFoo() and .getFoo() on as well > > on too. > > > > Another reason I dislike the project builder is that we really should > > not be using ${project} on its own in the mojos. That binds you to a > > non-trivial library in Maven core that limits the ability to change the > > version. More use of ${project.field} instead would be welcomed by me, > > and I feel is more in the original spirit of Mojos. > > > > As before, I think we need to start doing the tests and see where it > > leads us. And definitely need to criticially review the tests and try > > and find ways to make them simpler in code, and faster to write. > > > > >> At one point, we were doing away with private field injection and > > >> recommending setters, so it is how they'd be used. > > > > > > Hasn't happened yet. Not sure if that will happen as I'm not sure how > > > many Ant tasks are actually used outside of Ant. Given what we do in > > > promoting the creation of components that get wrapped by a Mojo to > > > expose them in Maven I'm not sure if there's much point to properties. > > > If that's what we decide then we can do that. I'm not fussed one way > or > > > the other. > > > > Ok, likewise. I think this is tied into the dependence on Maven objects > > instead of making them simple POJOs. Maybe it was putting the M in POJO > > that was the problem, rather than a lack of OJO in MOJO :) > > > > > But there are plugins that require a MavenProject and other parts of > the > > > POM not in the configuration for the plugin. > > > The IDEA plugin is one example that needs a POM so having the pom.xml > > > file actually be the source of a MavenProject I think would work. > There > > > is also the resources plugin which requires elements in the <build/> > > > element. I don't think we'll get away with just the <configuration/> > > > element for the plugin. > > > > If it needs to be a POM, it should be a POM, loaded with the project > > loader. If it is not a POM, it should not look like a POM, and just be > > the XML configuration fragment inserted into the instantiated Mojo. Or > > it could all be done in java code with a stub project and setters on the > > mojo. I'd like the alternative for both. > > > > > We've had this discussion before :-) They are not properties. We > happen > > > to abuse the field name being the same as the configuration element. > But > > > with a property the configuration element can be different then the > > > field name as the setter would intervene. > > > > Ok, I get confused on the terminology, sorry. But isn't it the case > > right now that if I add a setter to a mojo, that gets called instead? Or > > is that only if you add the property element to the configuration? > > > > I think this needs to be written down in the documentation, as I suspect > > nobody other than yourself knows how the setters truly work right now > > and it's probably clouding our perception of how to deal with them here. > > property="..." is not listed on > > http://maven.apache.org/developers/mojo-api-specification.html. > > > > > In how many cases in a Mojo can you only set one field and actually > make > > > it do anything? > > > > surefirePlugin.setForkMode( "once" ); > > surefirePlugin.setForkMode( "pertest" ); > > surefirePlugin.setForkMode( "none" ); > > > > I think there are plenty of examples. > > > > > I'm all for a Java way, but I think the pom.xml will be > > > the most thorough and actually correspond most accurately to real use. > > > Whatever kind of testing that actually is. > > > > That's fine. Let's allow both, and find out through experience. > > > > > In SuiteRunner the input and assertions were in the same file. Sure, > it > > > can't hurt to have both methods and I think we need a method in Java > > > simply because we will probably find holes in the declarative > structure > > > and that shouldn't stop you from testing. > > > > Ok, so we are generally in agreement. Summary at the top. > > > > Cheers, > > Brett > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > ___________________________________________________________________________ > Nouveau : t�l�phonez moins cher avec Yahoo! Messenger ! D�couvez les > tarifs exceptionnels pour appeler la France et l'international. > T�l�chargez sur http://fr.messenger.yahoo.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > -- jesse mcconnell jesseDOTmcconnellATgmailDOTcom
