+1 2006/9/4, Alex Blewitt <[EMAIL PROTECTED]>:
IMNSO it doesn't make sense to arbitrarily partition the tests based on a moniker, such as 'integration test', 'unit test', 'regression test' etc. For one thing, developers are generally not good at agreeing on the difference between them :-)If you've got fast and slow tests, then have a group for fast and slow tests. Then you can choose to just run the fast tests, and any automated build system can handle running the slow tests. Alex. On 04/09/06, Vladimir Ivanov <[EMAIL PROTECTED]> wrote: > On 9/4/06, Richard Liang <[EMAIL PROTECTED]> wrote: > > > > > > > > Mikhail Loenko wrote: > > > Hi Vladimir > > > > > > Could you please decribe for what purpose it will be used? > > > > > > I mean why one might have to either exclude or run only regression > > tests? > > > > If running all tests takes up much time, running all regression test > > (for one particular version) may be a convenient way to verify the > > correctness of bug-fixing. > > > Thanks Richard. It is exactly what I want to say :) > On the other hand may be all proposed groups need similar explanation (smt. > like use-case for group)? > > thanks, Vladimir > > Best regards, > > Richard. > > > > > > > > Thanks, > > > Mikhail > > > > > > 2006/8/30, Vladimir Ivanov <[EMAIL PROTECTED]>: > > >> On 8/30/06, Richard Liang <[EMAIL PROTECTED]> wrote: > > >> > > > >> > > > >> > > > >> > Vladimir Ivanov wrote: > > >> > > Also some tag for regression tests should be added. > > >> > Yes. Do you think we could annotate regression test as > > >> > *level.regression*? Thanks a lot. > > >> > > >> > > >> Yes, I do. While tests can have more than one group it will enough. > > >> thanks, Vladimir > > >> > > >> > > >> Richard > > >> > > thanks, Vladimir > > >> > > > > >> > > > > >> > > On 8/28/06, Richard Liang <[EMAIL PROTECTED]> wrote: > > >> > >> > > >> > >> > > >> > >> > > >> > >> Richard Liang wrote: > > >> > >> > Hello All, > > >> > >> > > > >> > >> > Now let's talk about the TestNG groups. I have read the related > > >> > >> > threads which posted by George, Vladimir Ivanov and Alexei > > >> Zakharov. > > >> > >> > All of them are good discussion about TestNG groups. > > >> > >> > > > >> > >> > IMHO, we may define Harmony test groups according the following > > 4 > > >> > >> > dimensions: > > >> > >> > > > >> > >> > 1) [Platform] os.any, os.<platform id> > > >> > >> > *os.any* - group of tests which pass on any platform. IMHO, > > >> most of > > >> > >> > our tests should be in this group. > > >> > >> > *os.<platform id>* - group of tests which are designed for one > > >> > >> > specific platform. A test may be in more than one of the > > >> groups. e.g > > >> > ., > > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"}) > > >> > >> > > > >> > >> > ** os.any and os.<platform id> are mutually exclusive, that > > >> is, > > >> > >> > tests in os.any group should not be in os.win.IA32. > > >> > >> > > > >> > >> > 2) [Test state] state.broken, state.broken.<platform id> > > >> > >> > *state.broken* - group of tests which fail on every platform, > > >> because > > >> > >> > of bugs of tests or implementation. We need to fix the bugs of > > >> tests > > >> > >> > or implementation to make them pass. > > >> > >> > *state.broken.<platform id>* - groups of test which only fail > > >> on one > > >> > >> > specific platform. A test may be in more than one of the > > >> groups. e.g > > >> > ., > > >> > >> > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64 > > "}) > > >> > >> > > > >> > >> > **state.broken.<platform id> group may be used as a > > >> convenient > > >> > way > > >> > >> > to indicate that a test is platform-specific. e.g., If we > > >> support 10 > > >> > >> > platforms, and one test are designed for 9 platforms except for > > >> > MacOS, > > >> > >> > instead of list 9 os.<platform id>, we can just use > > >> > state.broken.MacOS > > >> > >> > > > >> > >> > 3) [Test type] type.api , type.impl > > >> > >> > *type.api* - group of tests which are tests for APIs in the Java > > >> > >> > Specification > > >> > >> > *type.impl* - groups of tests which are tests for > > >> Harmony-specific > > >> > >> > implementation > > >> > >> > > > >> > >> > ** type.api and type.impl are also mutually exclusive. > > >> > >> > > > >> > >> > 4) [Test Level] level.unit, level.integration, level.system, > > >> > >> > level.stress, etc. (Levels of Test refer to the increase in > > >> > complexity > > >> > >> > as moving through test cycle. ) > > >> > >> > ** A test may be in more than one of the groups. > > >> > >> > ** In fact, some tests such as System tests are the > > >> verification > > >> > of > > >> > >> > the entire system. Maybe we'll put them into a separate > > project. > > >> > >> > e.g., harmony/enhanced/SVT (System Verification Test). > > >> > >> > > > >> > >> > If we want to run all the unit test for APIs on windows, we > > >> may use > > >> > >> > TestNG groups to select the tests: > > >> > >> > <groups> > > >> > >> > <run> > > >> > >> > <include name="os.any" /> > > >> > >> > <include name="type.api" /> > > >> > >> > <include name="os.win.IA32" /> > > >> > >> > <exclude name=" state.broken" /> > > >> > >> > <exclude name="state.broken.win.IA32" /> > > >> > >> > </run> > > >> > >> > </groups> > > >> > >> > > > >> > >> Hello All, > > >> > >> > > >> > >> I'm sorry. It seems that the example does not work. I will try to > > >> > figure > > >> > >> another example soon. ;-) > > >> > >> > > >> > >> Best regards, > > >> > >> Richard > > >> > >> > > > >> > >> > Well, I think our most of existing tests are in the groups of > > >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked TestNG > > >> to add > > >> > a > > >> > >> > new option "-groups" for its JUnitConverter which allow us to > > >> specify > > >> > >> > the test groups when migrate from JUnit test to TestNG test. > > >> > >> > > > >> > >> > Thanks for reading so far, and I will highly appreciate your > > >> comments > > >> > >> > or suggestion. ;-) > > >> > >> > > > >> > >> > > >> > >> -- > > >> > >> Richard Liang > > >> > >> China Software Development Lab, IBM > > >> > >> > > >> > >> > > >> > >> > > >> > >> > > >> --------------------------------------------------------------------- > > >> > >> Terms of use : http://incubator.apache.org/harmony/mailing.html > > >> > >> To unsubscribe, e-mail: > > >> [EMAIL PROTECTED] > > >> > >> For additional commands, e-mail: > > >> [EMAIL PROTECTED] > > >> > >> > > >> > >> > > >> > > > > >> > > > >> > -- > > >> > Richard Liang > > >> > China Software Development Lab, IBM > > >> > > > >> > > > >> > > > >> > --------------------------------------------------------------------- > > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html > > >> > To unsubscribe, e-mail: [EMAIL PROTECTED] > > >> > For additional commands, e-mail: > > [EMAIL PROTECTED] > > >> > > > >> > > > >> > > >> > > > > > > --------------------------------------------------------------------- > > > Terms of use : http://incubator.apache.org/harmony/mailing.html > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > -- > > Richard Liang > > China Software Development Lab, IBM > > > > > > > > --------------------------------------------------------------------- > > Terms of use : http://incubator.apache.org/harmony/mailing.html > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > --------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
