Nothing is wrong with TestNG and I agree TestNG and Maven/Surefire integration could be better, but there that is not all on Maven/Surefire, there are some issues with how TestNG supports integration as well.
The point I was making is that, currently, maven doesn't really support integration or functional testing in clear concise way like it does for unit testing. Personally, I like to break up tests into unit-tests, functional tests, small integration tests, and full scenario tests. There is no "easy" way to do this currently maven. That is why I started to use TestNG groups to allow me to only run certain tests at certain times. However, there are still a lot problems with this approach when you want to create fast builds that can provide feedback on tests and fast as possible. You might even have to break them out into separate projects and then you have run tests in different project to effectively test a single project. Sure, you can do a lot of different things to get around it but they all have some drawbacks, some of which Stephen already mentioned and some which I have mentioned. The FailSafe plugin attempts to solve some of the problems with the current support in as clean a way as possible. However, it only takes into account the integration tests. Whereas the wiki discussion had a suggestion of having functional tests as well and I like that. Jason -----Original Message----- From: Stevo Slavić [mailto:ssla...@gmail.com] Sent: Tuesday, May 26, 2009 6:01 PM To: Maven Users List Subject: Re: [ANN] Failsafe Maven Plugin 2.4.3-alpha-1 released. What's wrong with using TestNG? Why is it better to change Maven standard directory layout? TestNG works already and doesn't impose package or test class naming constraints. Btw, Surefire plugin support for TestNG could be better. Failsafe plugin is welcome but I'd prefer a single well maintained plugin (failsafe or surefire) that could handle running and verification of tests in both Maven test and integration-test/verify phases. Would it be possible to have Surefire/Failsafe plugin with 4 goals: "run-tests", "verify-tests", "test", and "report-only"? Test goal could in default execution, as current surefire test goal, be bound to test phase, including in it's execution both run-tests and verify-tests goals/tasks (like tasks in cobertura plugin). run-tests and verify-tests shouldn't be bound to any phase. Where and when needed one could configure two new surefire/failsafe executions, e.g. one with run-tests goal bound to integration-test phase, and another one with verify-tests goal bound to verify phase. run-tests should just run all configured tests, and shouldn't fail build regardles of test outcome, but should as side effect somehow share test execution status/data with other plugins/goals in a build, so verify-tests goal/task could use that information (not) to fail a build if/when verify-tests gets executed and report-only could just generate a report. Executing verify-tests and report-only would only be meaningful if run-tests gets executed ahead of them. Does this make any sense to others? Is it feasible with what Maven 2 currently provides? Regards, Stevo. On Wed, May 27, 2009 at 12:25 AM, Jason Chaffee <jason.chaf...@zilliontv.tv>wrote: > I really like the idea of breaking up unit, integration, and functional > tests. This is how I would prefer to break up my tests. I had to use TestNG > groups to do this effectively in the current scheme of things. > > Thanks for the link! > > Jason > > -----Original Message----- > From: Stephen Connolly [mailto:stephen.alan.conno...@gmail.com] > Sent: Tuesday, May 26, 2009 2:14 PM > To: Maven Users List > Cc: Maven Developers List > Subject: Re: [ANN] Failsafe Maven Plugin 2.4.3-alpha-1 released. > > 2009/5/26 Jason Chaffee <jason.chaf...@zilliontv.tv> > > > Yep, I concur with you a 100%. I think the solutions for this until now > > (FailSafe), were highly undesirable and caused many other issues. > > > > It might be something the Maven folks might want to think about in the > > future, adding support for integration testing in an more appropriate way > > (adding the maven dev list for this reason). > > > I know that there is already some proposals bobbing around, e.g. > > http://docs.codehaus.org/display/MAVEN/best+practices+-+testing+strategies > > > > > > > > > Regards, > > > > Jason > > > > > > > > -----Original Message----- > > From: Stephen Connolly [mailto:stephen.alan.conno...@gmail.com] > > Sent: Tuesday, May 26, 2009 1:08 PM > > To: Maven Users List > > Cc: Maven Users List > > Subject: Re: [ANN] Failsafe Maven Plugin 2.4.3-alpha-1 released. > > > > and you'd probably want an integration-test scope as well for > > dependencies used for integration testing, and you'd need to add an > > integrationTestSources element to the build, as well as > > integrationTestResources > > > > that is if you want to do things properly. > > > > and I am not opposed to such changes. the failsafe plugin provides a > > mostly clean solution *now*. I am fed up waiting for try... finally > > phases (which are why failing the build in the integration-test phase > > is *bad*)... I am fed up having a big mad configuration section with > > multiple executions in my maven-surefire-plugin plugin, which scares > > project newcomers, just the get integration tests running correctly... > > I am fed up with a separate module for integration tests (of only one > > module) because you only really see the failures when it comes time to > > release... > > > > with failsafe, it's part of the standard build for that module. > > > > if the developers want to skip the its > > > > mvn package > > or > > mvn verify -DskipITs > > > > if they just run mvn install or mvn deploy then the integration tests > > will have run. > > > > if they've bugs in unit tests > > > > mvn test -Dmaven.surefire.debug=... > > > > if they've bugs in integration tests > > > > mvn verify -Dmaven.failsafe.debug=... > > > > and all I'd added is a simple short snippet to the pom... no execution > > hacks, no profile hacks > > > > don't get me wrong, if somebody has a better solution *pretty please > > with catnip on top* show it to us *now* ;-) > > > > -Stephen > > > > Sent from my [rhymes with myPod] ;-) > > > > On 26 May 2009, at 20:33, Jason Chaffee <jason.chaf...@zilliontv.tv> > > wrote: > > > > > Yes, it would basically force Maven to adopt it as a standard and it > > > would might mean even adding a itest-compile phase, etc. > > > > > > Not sure if I prefer this idea over the current protocol, but I do > > > think the FailSafe plugin is very good and clever approach to > > > integration tests with the way things currently work in Maven. > > > > > > I am just throwing it out there as idea for the sake of discussion. > > > > > > Jason > > > > > > -----Original Message----- > > > From: Stephen Connolly [mailto:stephen.alan.conno...@gmail.com] > > > Sent: Tuesday, May 26, 2009 12:27 PM > > > To: Maven Users List > > > Cc: Maven Users List > > > Subject: Re: [ANN] Failsafe Maven Plugin 2.4.3-alpha-1 released. > > > > > > but that looses source folder config in ide setup, and hacks with > > > test- > > > compile > > > > > > Sent from my [rhymes with myPod] ;-) > > > > > > On 26 May 2009, at 20:22, Jason Chaffee <jason.chaf...@zilliontv.tv> > > > wrote: > > > > > >> One way around the excludes hack is to adopt a different directory > > >> structure for it tests and unit tests. For example > > >> > > >> src/test --> unit tests > > >> > > >> src/itest --> integration tests > > >> > > >> Jason > > >> > > >> -----Original Message----- > > >> From: Stephen Connolly [mailto:stephen.alan.conno...@gmail.com] > > >> Sent: Tuesday, May 26, 2009 12:12 PM > > >> To: Maven Users List > > >> Cc: Maven Users List > > >> Subject: Re: [ANN] Failsafe Maven Plugin 2.4.3-alpha-1 released. > > >> > > >> I am not opposed to doing so. the block for me is my lack of Apache > > >> commit access. > > >> > > >> there are valid arguments for keeping these as separate plugins > > >> though, eg the <excludes>none</excludes> hack that people used to > > >> configure surefire for multiple executions; debugging tests from an > > >> IDE; skipping one type of tests, etc > > >> > > >> but at the end of the day, if we can find a way to combine to one > > >> plugin, I'm fine with that > > >> > > >> Sent from my [rhymes with myPod] ;-) > > >> > > >> On 26 May 2009, at 19:58, Paul Benedict <pbened...@apache.org> wrote: > > >> > > >>> Will there be an effort to add the integration testing features to > > >>> the > > >>> original? I would like to not have multiple testing plugins. > > >>> > > >>> On Tue, May 26, 2009 at 1:54 PM, Stephen Connolly > > >>> <stephen.alan.conno...@gmail.com> wrote: > > >>>> use surefire for unit tests > > >>>> > > >>>> use failsafe if you need to set up a integration test environment > > >>>> and tear > > >>>> it back down again after the integration tests have ran > > >>>> > > >>>> -Stephen > > >>>> > > >>>> Sent from my [rhymes with myPod] ;-) > > >>>> > > >>>> On 26 May 2009, at 19:34, Wim Deblauwe <wim.debla...@gmail.com> > > >>>> wrote: > > >>>> > > >>>>> Is there an explanation somewhere on when to use surefire and when > > >>>>> this > > >>>>> failsafe plugin? What are the main differences? > > >>>>> > > >>>>> regards, > > >>>>> > > >>>>> Wim > > >>>>> > > >>>>> 2009/5/25 Stephen Connolly <stephen.alan.conno...@gmail.com> > > >>>>> > > >>>>>> The Mojo team is pleased to announce the release of the Failsafe > > >>>>>> Maven > > >>>>>> Plugin, version 2.4.3-alpha-1. > > >>>>>> > > >>>>>> The Failsafe Plugin is a fork of the Maven Surefire Plugin > > >>>>>> designed for > > >>>>>> running integration tests. It has the following goals. > > >>>>>> > > >>>>>> * failsafe:integration-test (with a default phase of integration- > > >>>>>> test) > > >>>>>> which by default uses surefire to execute all tests matching the > > >>>>>> pattern > > >>>>>> **/IT*.java, **/*IT.java and **/*ITCase.java. This goal should > > >>>>>> not fail > > >>>>>> the > > >>>>>> build, thereby allowing the post-integration-test phase to > > >>>>>> execute and > > >>>>>> tidy > > >>>>>> up the integration test environment. > > >>>>>> * failsafe:verify (with a default phase of verify) which checks > > >>>>>> the > > >>>>>> results of running the integration tests and fails the build if > > >>>>>> there > > >>>>>> were > > >>>>>> test failures. > > >>>>>> > > >>>>>> When using the Failsafe Maven Plugin with its default phase > > >>>>>> bindings, use > > >>>>>> > > >>>>>> mvn verify > > >>>>>> > > >>>>>> to execute the integration tests. > > >>>>>> > > >>>>>> To use this plugin in your projects, add the following to the > > >>>>>> project/build/plugins section of your pom.xml > > >>>>>> > > >>>>>> <project> > > >>>>>> [...] > > >>>>>> <build> > > >>>>>> [...] > > >>>>>> <plugins> > > >>>>>> [...] > > >>>>>> <plugin> > > >>>>>> <groupId>org.codehaus.mojo</groupId> > > >>>>>> <artifactId>failsafe-maven-plugin</artifactId> > > >>>>>> <version>2.4.3-alpha-1</version> > > >>>>>> <executions> > > >>>>>> <execution> > > >>>>>> <goals> > > >>>>>> <goal>integration-test</goal> > > >>>>>> <goal>verify</goal> > > >>>>>> </goals> > > >>>>>> </execution> > > >>>>>> </executions> > > >>>>>> </plugin> > > >>>>>> [...] > > >>>>>> </plugins> > > >>>>>> [...] > > >>>>>> </build> > > >>>>>> [...] > > >>>>>> </project> > > >>>>>> > > >>>>>> The artifacts have been deployed to the mojo repository and will > > >>>>>> be > > >>>>>> mirrored > > >>>>>> to central within the next 24 hours. > > >>>>>> > > >>>>>> The Mojo Team. > > >>>>>> > > >>>> > > >>>> --- > > >>>> ------------------------------------------------------------------ > > >>>> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > > >>>> For additional commands, e-mail: users-h...@maven.apache.org > > >>>> > > >>>> > > >>> > > >>> --- > > >>> ------------------------------------------------------------------ > > >>> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > > >>> For additional commands, e-mail: users-h...@maven.apache.org > > >>> > > >> > > >> --------------------------------------------------------------------- > > >> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > > >> For additional commands, e-mail: users-h...@maven.apache.org > > >> > > >> > > >> --------------------------------------------------------------------- > > >> To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > > >> For additional commands, e-mail: users-h...@maven.apache.org > > >> > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > > > For additional commands, e-mail: users-h...@maven.apache.org > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > > > For additional commands, e-mail: users-h...@maven.apache.org > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > > For additional commands, e-mail: users-h...@maven.apache.org > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > > For additional commands, e-mail: users-h...@maven.apache.org > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@maven.apache.org > For additional commands, e-mail: users-h...@maven.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@maven.apache.org For additional commands, e-mail: users-h...@maven.apache.org
