IMNSO it doesn't make sense to arbitrarily partition the tests based
on a moniker, such as 'integration test', 'unit test', 'regression
test' etc. For one thing, developers are generally not good at
agreeing on the difference between them :-)
If you've got fast and slow tests, then have a group for fast and slow
tests. Then you can choose to just run the fast tests, and any
automated build system can handle running the slow tests.
Alex.
On 04/09/06, Vladimir Ivanov <[EMAIL PROTECTED]> wrote:
On 9/4/06, Richard Liang <[EMAIL PROTECTED]> wrote:
>
>
>
> Mikhail Loenko wrote:
> > Hi Vladimir
> >
> > Could you please decribe for what purpose it will be used?
> >
> > I mean why one might have to either exclude or run only regression
> tests?
>
> If running all tests takes up much time, running all regression test
> (for one particular version) may be a convenient way to verify the
> correctness of bug-fixing.
Thanks Richard. It is exactly what I want to say :)
On the other hand may be all proposed groups need similar explanation (smt.
like use-case for group)?
thanks, Vladimir
Best regards,
> Richard.
>
> >
> > Thanks,
> > Mikhail
> >
> > 2006/8/30, Vladimir Ivanov <[EMAIL PROTECTED]>:
> >> On 8/30/06, Richard Liang <[EMAIL PROTECTED]> wrote:
> >> >
> >> >
> >> >
> >> > Vladimir Ivanov wrote:
> >> > > Also some tag for regression tests should be added.
> >> > Yes. Do you think we could annotate regression test as
> >> > *level.regression*? Thanks a lot.
> >>
> >>
> >> Yes, I do. While tests can have more than one group it will enough.
> >> thanks, Vladimir
> >>
> >>
> >> Richard
> >> > > thanks, Vladimir
> >> > >
> >> > >
> >> > > On 8/28/06, Richard Liang <[EMAIL PROTECTED]> wrote:
> >> > >>
> >> > >>
> >> > >>
> >> > >> Richard Liang wrote:
> >> > >> > Hello All,
> >> > >> >
> >> > >> > Now let's talk about the TestNG groups. I have read the related
> >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
> >> Zakharov.
> >> > >> > All of them are good discussion about TestNG groups.
> >> > >> >
> >> > >> > IMHO, we may define Harmony test groups according the following
> 4
> >> > >> > dimensions:
> >> > >> >
> >> > >> > 1) [Platform] os.any, os.<platform id>
> >> > >> > *os.any* - group of tests which pass on any platform. IMHO,
> >> most of
> >> > >> > our tests should be in this group.
> >> > >> > *os.<platform id>* - group of tests which are designed for one
> >> > >> > specific platform. A test may be in more than one of the
> >> groups. e.g
> >> > .,
> >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> >> > >> >
> >> > >> > ** os.any and os.<platform id> are mutually exclusive, that
> >> is,
> >> > >> > tests in os.any group should not be in os.win.IA32.
> >> > >> >
> >> > >> > 2) [Test state] state.broken, state.broken.<platform id>
> >> > >> > *state.broken* - group of tests which fail on every platform,
> >> because
> >> > >> > of bugs of tests or implementation. We need to fix the bugs of
> >> tests
> >> > >> > or implementation to make them pass.
> >> > >> > *state.broken.<platform id>* - groups of test which only fail
> >> on one
> >> > >> > specific platform. A test may be in more than one of the
> >> groups. e.g
> >> > .,
> >> > >> > @Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64
> "})
> >> > >> >
> >> > >> > **state.broken.<platform id> group may be used as a
> >> convenient
> >> > way
> >> > >> > to indicate that a test is platform-specific. e.g., If we
> >> support 10
> >> > >> > platforms, and one test are designed for 9 platforms except for
> >> > MacOS,
> >> > >> > instead of list 9 os.<platform id>, we can just use
> >> > state.broken.MacOS
> >> > >> >
> >> > >> > 3) [Test type] type.api , type.impl
> >> > >> > *type.api* - group of tests which are tests for APIs in the Java
> >> > >> > Specification
> >> > >> > *type.impl* - groups of tests which are tests for
> >> Harmony-specific
> >> > >> > implementation
> >> > >> >
> >> > >> > ** type.api and type.impl are also mutually exclusive.
> >> > >> >
> >> > >> > 4) [Test Level] level.unit, level.integration, level.system,
> >> > >> > level.stress, etc. (Levels of Test refer to the increase in
> >> > complexity
> >> > >> > as moving through test cycle. )
> >> > >> > ** A test may be in more than one of the groups.
> >> > >> > ** In fact, some tests such as System tests are the
> >> verification
> >> > of
> >> > >> > the entire system. Maybe we'll put them into a separate
> project.
> >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
> >> > >> >
> >> > >> > If we want to run all the unit test for APIs on windows, we
> >> may use
> >> > >> > TestNG groups to select the tests:
> >> > >> > <groups>
> >> > >> > <run>
> >> > >> > <include name="os.any" />
> >> > >> > <include name="type.api" />
> >> > >> > <include name="os.win.IA32" />
> >> > >> > <exclude name=" state.broken" />
> >> > >> > <exclude name="state.broken.win.IA32" />
> >> > >> > </run>
> >> > >> > </groups>
> >> > >> >
> >> > >> Hello All,
> >> > >>
> >> > >> I'm sorry. It seems that the example does not work. I will try to
> >> > figure
> >> > >> another example soon. ;-)
> >> > >>
> >> > >> Best regards,
> >> > >> Richard
> >> > >> >
> >> > >> > Well, I think our most of existing tests are in the groups of
> >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked TestNG
> >> to add
> >> > a
> >> > >> > new option "-groups" for its JUnitConverter which allow us to
> >> specify
> >> > >> > the test groups when migrate from JUnit test to TestNG test.
> >> > >> >
> >> > >> > Thanks for reading so far, and I will highly appreciate your
> >> comments
> >> > >> > or suggestion. ;-)
> >> > >> >
> >> > >>
> >> > >> --
> >> > >> Richard Liang
> >> > >> China Software Development Lab, IBM
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >> ---------------------------------------------------------------------
> >> > >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> > >> To unsubscribe, e-mail:
> >> [EMAIL PROTECTED]
> >> > >> For additional commands, e-mail:
> >> [EMAIL PROTECTED]
> >> > >>
> >> > >>
> >> > >
> >> >
> >> > --
> >> > Richard Liang
> >> > China Software Development Lab, IBM
> >> >
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> > To unsubscribe, e-mail: [EMAIL PROTECTED]
> >> > For additional commands, e-mail:
> [EMAIL PROTECTED]
> >> >
> >> >
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > For additional commands, e-mail: [EMAIL PROTECTED]
> >
> >
>
> --
> Richard Liang
> China Software Development Lab, IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
