>-----Original Message-----
>From: Richard Liang [mailto:[EMAIL PROTECTED]
>Sent: Monday, September 04, 2006 10:56 AM
>To: harmony-dev@incubator.apache.org
>Subject: Re: [classlib][TestNG] groups of Harmony test
>
>
>
>Mikhail Loenko wrote:
>> Hi Vladimir
>>
>> Could you please decribe for what purpose it will be used?
>>
>> I mean why one might have to either exclude or run only regression
tests?
>
>If running all tests takes up much time, running all regression test
>(for one particular version) may be a convenient way to verify the
>correctness of bug-fixing.
I can be used for informational purposes as well. If the test fails, you
need simply re-open the associated bug provided the bug number can be
easily found.
Regards,
Alexey.
>
>Best regards,
>Richard.
>
>>
>> Thanks,
>> Mikhail
>>
>> 2006/8/30, Vladimir Ivanov <[EMAIL PROTECTED]>:
>>> On 8/30/06, Richard Liang <[EMAIL PROTECTED]> wrote:
>>> >
>>> >
>>> >
>>> > Vladimir Ivanov wrote:
>>> > > Also some tag for regression tests should be added.
>>> > Yes. Do you think we could annotate regression test as
>>> > *level.regression*? Thanks a lot.
>>>
>>>
>>> Yes, I do. While tests can have more than one group it will enough.
>>> thanks, Vladimir
>>>
>>>
>>> Richard
>>> > > thanks, Vladimir
>>> > >
>>> > >
>>> > > On 8/28/06, Richard Liang <[EMAIL PROTECTED]> wrote:
>>> > >>
>>> > >>
>>> > >>
>>> > >> Richard Liang wrote:
>>> > >> > Hello All,
>>> > >> >
>>> > >> > Now let's talk about the TestNG groups. I have read the
related
>>> > >> > threads which posted by George, Vladimir Ivanov and Alexei
>>> Zakharov.
>>> > >> > All of them are good discussion about TestNG groups.
>>> > >> >
>>> > >> > IMHO, we may define Harmony test groups according the
following 4
>>> > >> > dimensions:
>>> > >> >
>>> > >> > 1) [Platform] os.any, os.<platform id>
>>> > >> > *os.any* - group of tests which pass on any platform. IMHO,
>>> most of
>>> > >> > our tests should be in this group.
>>> > >> > *os.<platform id>* - group of tests which are designed for
one
>>> > >> > specific platform. A test may be in more than one of the
>>> groups. e.g
>>> > .,
>>> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
>>> > >> >
>>> > >> > ** os.any and os.<platform id> are mutually exclusive,
that
>>> is,
>>> > >> > tests in os.any group should not be in os.win.IA32.
>>> > >> >
>>> > >> > 2) [Test state] state.broken, state.broken.<platform id>
>>> > >> > *state.broken* - group of tests which fail on every platform,
>>> because
>>> > >> > of bugs of tests or implementation. We need to fix the bugs
of
>>> tests
>>> > >> > or implementation to make them pass.
>>> > >> > *state.broken.<platform id>* - groups of test which only fail
>>> on one
>>> > >> > specific platform. A test may be in more than one of the
>>> groups. e.g
>>> > .,
>>> > >> > @Test(groups={"state.broken.linux.IA32",
"os.broken.linux.IA64"})
>>> > >> >
>>> > >> > **state.broken.<platform id> group may be used as a
>>> convenient
>>> > way
>>> > >> > to indicate that a test is platform-specific. e.g., If we
>>> support 10
>>> > >> > platforms, and one test are designed for 9 platforms except
for
>>> > MacOS,
>>> > >> > instead of list 9 os.<platform id>, we can just use
>>> > state.broken.MacOS
>>> > >> >
>>> > >> > 3) [Test type] type.api , type.impl
>>> > >> > *type.api* - group of tests which are tests for APIs in the
Java
>>> > >> > Specification
>>> > >> > *type.impl* - groups of tests which are tests for
>>> Harmony-specific
>>> > >> > implementation
>>> > >> >
>>> > >> > ** type.api and type.impl are also mutually exclusive.
>>> > >> >
>>> > >> > 4) [Test Level] level.unit, level.integration, level.system,
>>> > >> > level.stress, etc. (Levels of Test refer to the increase in
>>> > complexity
>>> > >> > as moving through test cycle. )
>>> > >> > ** A test may be in more than one of the groups.
>>> > >> > ** In fact, some tests such as System tests are the
>>> verification
>>> > of
>>> > >> > the entire system. Maybe we'll put them into a separate
project.
>>> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
>>> > >> >
>>> > >> > If we want to run all the unit test for APIs on windows, we
>>> may use
>>> > >> > TestNG groups to select the tests:
>>> > >> > <groups>
>>> > >> > <run>
>>> > >> > <include name="os.any" />
>>> > >> > <include name="type.api" />
>>> > >> > <include name="os.win.IA32" />
>>> > >> > <exclude name=" state.broken" />
>>> > >> > <exclude name="state.broken.win.IA32" />
>>> > >> > </run>
>>> > >> > </groups>
>>> > >> >
>>> > >> Hello All,
>>> > >>
>>> > >> I'm sorry. It seems that the example does not work. I will try
to
>>> > figure
>>> > >> another example soon. ;-)
>>> > >>
>>> > >> Best regards,
>>> > >> Richard
>>> > >> >
>>> > >> > Well, I think our most of existing tests are in the groups of
>>> > >> > {"os.any", "type.api", "level.unit"}, and I have asked TestNG
>>> to add
>>> > a
>>> > >> > new option "-groups" for its JUnitConverter which allow us to
>>> specify
>>> > >> > the test groups when migrate from JUnit test to TestNG test.
>>> > >> >
>>> > >> > Thanks for reading so far, and I will highly appreciate your
>>> comments
>>> > >> > or suggestion. ;-)
>>> > >> >
>>> > >>
>>> > >> --
>>> > >> Richard Liang
>>> > >> China Software Development Lab, IBM
>>> > >>
>>> > >>
>>> > >>
>>> > >>
>>>
---------------------------------------------------------------------
>>> > >> Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> > >> To unsubscribe, e-mail:
>>> [EMAIL PROTECTED]
>>> > >> For additional commands, e-mail:
>>> [EMAIL PROTECTED]
>>> > >>
>>> > >>
>>> > >
>>> >
>>> > --
>>> > Richard Liang
>>> > China Software Development Lab, IBM
>>> >
>>> >
>>> >
>>> >
---------------------------------------------------------------------
>>> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>>> > To unsubscribe, e-mail:
[EMAIL PROTECTED]
>>> > For additional commands, e-mail:
[EMAIL PROTECTED]
>>> >
>>> >
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail:
[EMAIL PROTECTED]
>>
>>
>
>--
>Richard Liang
>China Software Development Lab, IBM
>
>
>
>---------------------------------------------------------------------
>Terms of use : http://incubator.apache.org/harmony/mailing.html
>To unsubscribe, e-mail: [EMAIL PROTECTED]
>For additional commands, e-mail: [EMAIL PROTECTED]
--
Alexey A. Ivanov
Intel Middleware Product Division
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
