I sent my reply this afternoon, but I have not received it. (it seems there
is something wrong with my smtp server). So I send it again. :-)
On 9/4/06, Alex Blewitt <[EMAIL PROTECTED]> wrote:
IMNSO it doesn't make sense to arbitrarily partition the tests based
on a moniker, such as 'integration test', 'unit test', 'regression
test' etc. For one thing, developers are generally not good at
agreeing on the difference between them :-)
This is really a problem, however it might be simpler than we imagine. We
are open to any discussion. ;-) Anyway, developers are required to write
unit tests.
If you've got fast and slow tests, then have a group for fast and slow
tests. Then you can choose to just run the fast tests, and any
automated build system can handle running the slow tests.
IMHO, "fast or slow" may not be the key point. The question is whether we
have any requirements to run only the regression tests.
Best regards,
Richard
Alex.
On 04/09/06, Vladimir Ivanov <[EMAIL PROTECTED]> wrote:
> On 9/4/06, Richard Liang <[EMAIL PROTECTED]> wrote:
> >
> >
> >
> > Mikhail Loenko wrote:
> > > Hi Vladimir
> > >
> > > Could you please decribe for what purpose it will be used?
> > >
> > > I mean why one might have to either exclude or run only regression
> > tests?
> >
> > If running all tests takes up much time, running all regression test
> > (for one particular version) may be a convenient way to verify the
> > correctness of bug-fixing.
>
>
> Thanks Richard. It is exactly what I want to say :)
> On the other hand may be all proposed groups need similar explanation
(smt.
> like use-case for group)?
>
> thanks, Vladimir
>
> Best regards,
> > Richard.
> >
> > >
> > > Thanks,
> > > Mikhail
> > >
> > > 2006/8/30, Vladimir Ivanov <[EMAIL PROTECTED]>:
> > >> On 8/30/06, Richard Liang <[EMAIL PROTECTED]> wrote:
> > >> >
> > >> >
> > >> >
> > >> > Vladimir Ivanov wrote:
> > >> > > Also some tag for regression tests should be added.
> > >> > Yes. Do you think we could annotate regression test as
> > >> > *level.regression*? Thanks a lot.
> > >>
> > >>
> > >> Yes, I do. While tests can have more than one group it will enough.
> > >> thanks, Vladimir
> > >>
> > >>
> > >> Richard
> > >> > > thanks, Vladimir
> > >> > >
> > >> > >
> > >> > > On 8/28/06, Richard Liang <[EMAIL PROTECTED]> wrote:
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >> Richard Liang wrote:
> > >> > >> > Hello All,
> > >> > >> >
> > >> > >> > Now let's talk about the TestNG groups. I have read the
related
> > >> > >> > threads which posted by George, Vladimir Ivanov and Alexei
> > >> Zakharov.
> > >> > >> > All of them are good discussion about TestNG groups.
> > >> > >> >
> > >> > >> > IMHO, we may define Harmony test groups according the
following
> > 4
> > >> > >> > dimensions:
> > >> > >> >
> > >> > >> > 1) [Platform] os.any, os.<platform id>
> > >> > >> > *os.any* - group of tests which pass on any platform. IMHO,
> > >> most of
> > >> > >> > our tests should be in this group.
> > >> > >> > *os.<platform id>* - group of tests which are designed for
one
> > >> > >> > specific platform. A test may be in more than one of the
> > >> groups. e.g
> > >> > .,
> > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
> > >> > >> >
> > >> > >> > ** os.any and os.<platform id> are mutually exclusive,
that
> > >> is,
> > >> > >> > tests in os.any group should not be in os.win.IA32.
> > >> > >> >
> > >> > >> > 2) [Test state] state.broken, state.broken.<platform id>
> > >> > >> > *state.broken* - group of tests which fail on every
platform,
> > >> because
> > >> > >> > of bugs of tests or implementation. We need to fix the bugs
of
> > >> tests
> > >> > >> > or implementation to make them pass.
> > >> > >> > *state.broken.<platform id>* - groups of test which only
fail
> > >> on one
> > >> > >> > specific platform. A test may be in more than one of the
> > >> groups. e.g
> > >> > .,
> > >> > >> > @Test(groups={"state.broken.linux.IA32", "
os.broken.linux.IA64
> > "})
> > >> > >> >
> > >> > >> > **state.broken.<platform id> group may be used as a
> > >> convenient
> > >> > way
> > >> > >> > to indicate that a test is platform-specific. e.g., If we
> > >> support 10
> > >> > >> > platforms, and one test are designed for 9 platforms except
for
> > >> > MacOS,
> > >> > >> > instead of list 9 os.<platform id>, we can just use
> > >> > state.broken.MacOS
> > >> > >> >
> > >> > >> > 3) [Test type] type.api , type.impl
> > >> > >> > *type.api* - group of tests which are tests for APIs in the
Java
> > >> > >> > Specification
> > >> > >> > *type.impl* - groups of tests which are tests for
> > >> Harmony-specific
> > >> > >> > implementation
> > >> > >> >
> > >> > >> > ** type.api and type.impl are also mutually exclusive.
> > >> > >> >
> > >> > >> > 4) [Test Level] level.unit, level.integration, level.system,
> > >> > >> > level.stress, etc. (Levels of Test refer to the increase in
> > >> > complexity
> > >> > >> > as moving through test cycle. )
> > >> > >> > ** A test may be in more than one of the groups.
> > >> > >> > ** In fact, some tests such as System tests are the
> > >> verification
> > >> > of
> > >> > >> > the entire system. Maybe we'll put them into a separate
> > project.
> > >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
> > >> > >> >
> > >> > >> > If we want to run all the unit test for APIs on windows, we
> > >> may use
> > >> > >> > TestNG groups to select the tests:
> > >> > >> > <groups>
> > >> > >> > <run>
> > >> > >> > <include name="os.any" />
> > >> > >> > <include name="type.api" />
> > >> > >> > <include name="os.win.IA32" />
> > >> > >> > <exclude name=" state.broken" />
> > >> > >> > <exclude name="state.broken.win.IA32" />
> > >> > >> > </run>
> > >> > >> > </groups>
> > >> > >> >
> > >> > >> Hello All,
> > >> > >>
> > >> > >> I'm sorry. It seems that the example does not work. I will try
to
> > >> > figure
> > >> > >> another example soon. ;-)
> > >> > >>
> > >> > >> Best regards,
> > >> > >> Richard
> > >> > >> >
> > >> > >> > Well, I think our most of existing tests are in the groups
of
> > >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked
TestNG
> > >> to add
> > >> > a
> > >> > >> > new option "-groups" for its JUnitConverter which allow us
to
> > >> specify
> > >> > >> > the test groups when migrate from JUnit test to TestNG test.
> > >> > >> >
> > >> > >> > Thanks for reading so far, and I will highly appreciate your
> > >> comments
> > >> > >> > or suggestion. ;-)
> > >> > >> >
> > >> > >>
> > >> > >> --
> > >> > >> Richard Liang
> > >> > >> China Software Development Lab, IBM
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >>
> > >>
---------------------------------------------------------------------
> > >> > >> Terms of use :
http://incubator.apache.org/harmony/mailing.html
> > >> > >> To unsubscribe, e-mail:
> > >> [EMAIL PROTECTED]
> > >> > >> For additional commands, e-mail:
> > >> [EMAIL PROTECTED]
> > >> > >>
> > >> > >>
> > >> > >
> > >> >
> > >> > --
> > >> > Richard Liang
> > >> > China Software Development Lab, IBM
> > >> >
> > >> >
> > >> >
> > >> >
---------------------------------------------------------------------
> > >> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > >> > To unsubscribe, e-mail:
[EMAIL PROTECTED]
> > >> > For additional commands, e-mail:
> > [EMAIL PROTECTED]
> > >> >
> > >> >
> > >>
> > >>
> > >
> > >
---------------------------------------------------------------------
> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > > For additional commands, e-mail:
[EMAIL PROTECTED]
> > >
> > >
> >
> > --
> > Richard Liang
> > China Software Development Lab, IBM
> >
> >
> >
> > ---------------------------------------------------------------------
> > Terms of use : http://incubator.apache.org/harmony/mailing.html
> > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > For additional commands, e-mail: [EMAIL PROTECTED]
> >
> >
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Richard Liang
China Software Development Lab, IBM
